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Vorwort des Herausgebers 


Die Fahrzeugtechnik ist kontinuierlich Veränderungen unterworfen. Klima- 
wandel, die Verknappung einiger für Fahrzeugbau und -betrieb benötigter 
Rohstoffe, globaler Wettbewerb, gesellschaftlicher Wandel und das rapide 
Wachstum großer Städte erfordern neue Mobilitätslösungen, die vielfach eine 
Neudefinition des Fahrzeugs erforderlich machen. Die Forderungen nach 
Steigerung der Energieeffizienz, Emissionsreduktion, erhöhter Fahr- und Ar- 
beitssicherheit, Benutzerfreundlichkeit und angemessenen Kosten finden ihre 
Antworten nicht aus der singulären Verbesserung einzelner technischer Ele- 
mente, sondern benötigen Systemverständnis und eine domänenübergreifende 
Optimierung der Lösungen. 


Hierzu will die Karlsruher Schriftenreihe für Fahrzeugsystemtechnik einen 
Beitrag leisten. Für die Fahrzeuggattungen Pkw, Nfz, Mobile Arbeits- 
maschinen und Bahnfahrzeuge werden Forschungsarbeiten vorgestellt, die 
Fahrzeugsystemtechnik auf vier Ebenen beleuchten: das Fahrzeug als kom- 
plexes mechatronisches System, die Fahrer-Fahrzeug-Interaktion, das Fahrzeug 
in Verkehr und Infrastruktur sowie das Fahrzeug in Gesellschaft und Umwelt. 


Artund Zustand von Fahrbahnen sind bedeutsam für Verkehrssicherheit, unge- 
hinderten Verkehrs-fluss und Energiebedarf der Fahrzeuge. Üblicherweise 
werden in Deutschland Straßen und Brückenbauwerke in festen Intervallen 
von vier bzw. sechs Jahren auf Ihren Erhaltungszustand über-prüft. Dazu 
werden spezielle, aufwändige und kostenintensive Prüfverfahren eingesetzt. 
Neben den hohen Kosten der Prüfung ist ein erheblicher Nachteil die lange 
Zeitdauer zwischen aufeinanderfolgenden Prüfungen, so dass das Wissen über 
den Fahrbahnzustand meist nicht aktuell ist. Eine flächendeckende Karte der 
verbauten Fahrbahntypen gibt es gegenwärtig nicht. 


Die vorliegende Arbeit will hier Abhilfe schaffen. Es wird ein Verfahren 
vorgeschlagen, bei dem eine einfach in Kraftfahrzeuge integrierbare Messtech- 
nik über die Erfassung von Geräuschen und Schwingungen, die durch den 


Vorwort des Herausgebers 


Reifen-Fahrbahn-Rollkontakt am Fahrzeug entstehen, Art und Zustand der 
Fahrbahnen ermittelt. Dazu werden aus den Signalen charakteristische Muster 
aus Werten geeigneter Kenngrößen mittels verschiedener Methoden ermittelt. 
Es findet dabei eine Fusion der Daten verschiedener Fahrzeuge statt, was die 
Erkennungsrate deutlich erhöht und damit den messtechnischen Aufwand am 
einzelnen Fahrzeug gering hält. Eine mit dem vorgeschlagenen Messsystem 
ausgestattete Fahrzeugflotte kann stochastisch die gesamte Oberfläche aller 
Fahrbahnen erfassen und liefert so über die Zeit eine vollständige Karte. 


Im Ergebnis liefert dies die aktuelle Information über Fahrbahnart und —zus- 
tand sowie zu Änderungen über der Zeit. Es erlaubt ein situationsgerechtes 
Fahrbahnerhaltungsmanagement und die Erstellung von Zustandsprognosen, 
mit denen sich Instandhaltungskosten und volkswirtschaftliche Folgekosten 
durch Staus und Umleitungen reduzieren lassen. Die gewonnene Datenbasis 
erlaubt weiterhin die Verbesserung von 3D-Karten für assistiertes oder au- 
tomatisches Fahren und damit eine genauere Vorhersage von Bremsweg, Kur- 
vengrenzgeschwindigkeit, Rollwiderstand oder auch Rollgeräuschemission. 


Karlsruhe, 15.11.2019 Prof. Dr. rer. nat. Frank Gauterin 


Abstract 


The dissertation provides novel methods to process inertial sensor and acoustic 
sensor data forroad condition estimation and monitoring with application in ve- 
hicles, which serve as sensor platforms. Furthermore, methods are introduced 
to combine the results from various vehicles for a more reliable estimation. 


A systematic, continuous, and comprehensive monitoring of existing roads 
is becoming increasingly important. Yet, current practice is laborious and 
time-consuming as most steps of the process are done manually. 


A novel measuring device for vehicles is developed to autonomously acquire 
a large set of real world data on public roads. In addition, an extended sim- 
ulation approach is presented to record data under controlled conditions to 
investigate the impact of parameter variations. Furthermore, the methods for 
data processing are implemented in a user friendly Matlab toolbox to allow an 
automatic evaluation. 


The quantitative and visualized results show that the estimation of road uneven- 
ness and roughness accord with the ground truth. The prediction accuracy of 
road attributes, such as potholes, and road surfaces, such as damaged concrete 
or asphalt, yield 81.0 % and 96.1 % on average. The investigation of combina- 
tion strategies shows that multiple vehicle fusion incorporating the precision 
matrix from the vehicle classifiers increase the confidence level of the final 
estimation. The normalized mean square error of the regression function to 
estimate road roughness is 4.5 %o on average. 


Overall, the developed methods can improve or partially substitute the current 
practice to evaluate the road infrastructure. This work contributes concepts 
and methods to allow an automatic and comprehensive road condition mon- 
itoring. The data processing methods can run on electronic control units of 
modern vehicles, which are equipped with corresponding sensors. However, 
the presented system consisting of the developed measuring device and Matlab 
toolbox can also be applied on vehicle fleets as it is. 


Kurzfassung 


Diese Arbeit bietet neuartige Methoden zur Verarbeitung von Inertialsensor- 
und akustischen Sensordaten für die Straßenzustandsabschätzung und - 
überwachung mit Anwendung in Fahrzeugen, die als Sensorplattform dienen. 
Darüber hinaus werden Methoden eingeführt, um die Ergebnisse mehrerer 
Fahrzeuge für eine robuste Schätzung zu kombinieren. 


Eine systematische, kontinuierliche und umfassende Überwachung bestehen- 
der Straßen wird immer wichtiger. Die derzeitige Praxis ist jedoch mühsam 
und zeitaufwendig, da die meisten Schritte des Prozesses manuell durchgeführt 
werden. 


Ein neuartiges Messgerät für Fahrzeuge wurde entwickelt, um autonom eine 
große Menge an realen Daten auf öffentlichen Straßen zu erfassen. Darüber 
hinaus wird ein Simulationsansatz vorgestellt, um Daten unter kontrollierten 
Bedingungen aufzuzeichnen und die Auswirkungen von Parametervariationen 
zu untersuchen. Des Weiteren sind die Methoden zur Datenverarbeitung in 
einer benutzerfreundlichen Matlab-Toolbox implementiert, um eine automati- 
sche Auswertung zu ermöglichen. 


Die quantitativen und visualisierten Ergebnisse zeigen, dass die Abschätzung 
von Straßenunebenheiten und -rauheiten mit den tatsächlichen Gegebenhei- 
ten übereinstimmen. Die Vorhersagegenauigkeit von Attributen der Straße wie 
Schlaglöchern und Straßenoberflächen wie beschädigtem Beton oder Asphalt 
liegt im Durchschnitt bei 81,0 % bzw. 96,1 9. Die Untersuchung von Kombi- 
nationsstrategien zeigt, dass die Fusion mehrerer Fahrzeuge mit der Präzisions- 
matrix der Fahrzeugklassifikatoren die Sicherheit der endgültigen Schätzung 
erhöht. Der normalisierte mittlere quadratische Fehler der Regressionsfunktion 
zur Schätzung der Straßenrauhigkeit beträgt im Durchschnitt 4,5 %o. 


Insgesamt können die entwickelten Methoden die bisherigen Verfahren zur 
Bewertung der Straßeninfrastruktur verbessern oder teilweise ersetzen. Die 
vorliegende Arbeit trägt Konzepte und Methoden bei, um eine automatische 
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Kurzfassung 


und umfassende Straßenzustandsüberwachung zu ermöglichen. Die Datenver- 
arbeitung kann auf elektronischen Steuergeräten moderner Fahrzeuge erfolgen, 
die mit entsprechenden Sensoren ausgestattet sind. Das vorgestellte System, vor 
allem bestehend aus einem entwickelten Messgerät und einer Matlab-Toolbox, 
kann aber auch problemlos in seiner jetzigen Form auf Fahrzeugflotten ange- 
wendet werden. 
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1 Introduction 


Outline: This chapter gives the motivation in the first section. Section 1.2 
explains the causes for a degradation of road condition, the effects of road 
defects, and shows the necessity of anovel real-time road condition monitoring. 
The current process for road condition monitoring is described in Section 1.3 
and the following Section 1.4 reviews related literature of automated and 
mobile road condition monitoring. Based on these sections open problems are 
deduced (Section 1.5) and the aim and research question of this dissertation 
are given in Section 1.6 


1.1 Motivation 


The wealth pyramids of modern economies and studies show that the founda- 
tion of prosperity, productivity, growth, and social wellbeing are infrastructure 
and mobility [1]. 


In Germany, road transport is the main source of traffic volume and the back- 
bone of transport infrastructure for citizens and the economy [2]. However, 
the state of the road network deteriorates noticeably in many regions of Ger- 
many [3]. The road infrastructure is subject to permanent stress and needs 
to be repaired or renewed in order to ensure the substance and utility value. 
The degradation of the roadway condition results in a multitude of undesirable 
factors which burden vehicle users and third parties. 


Current practice of road condition monitoring is laborious and time-consuming 
as most steps of the process are done manually [4]. The steps involve the 
collection of data, the identification and assessment of defects based on the 
collected data, and the calculation of a road condition index for road segments 
with a specific length. Few countries have expensive specialized vehicles 
for an automated data collection whereas the monitoring is only scheduled 
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in fixed intervals of 1 to 4 years [5]. In most of the countries, the data is 
collected manually by inspectors, who pass the road infrastructure and record 
road defects, their location, type of distress, and severity level, among others. 
The evaluation of the road segments and defects is often subjective as it is 
based on the experience of the technician. 


At current practice defects are unlikely to be comprehensively identified in 
early stages, when repairs are more cost-efficient. Due to the decline in build- 
ing new roads and lack of financial resources, a systematic, continuous and 
comprehensive monitoring of existing roads is becoming increasingly impor- 
tant. With this approach, the functionality of the roads could be maintained in 
an economical and environmentally compatible way. 


1.2 Road condition 


The road profile is mainly characterized by the texture wavelength A [6]. The 
wavelengths physically illustrate the different lengths of periodically repeating 
structures of the profile. The road texture is categorized as microtexture with 
a wavelength of less than 0.5mm, as macrotexture with wavelengths from 
0.5 to 50 mm, and mega texture with wavelenghts from 50 mm to 0.5 m [6]. 
The profile texture with wavelengths from 0.5 m to 50 m is referred to as 
unevenness. 


The road profile is overlaid by road discontinuities, which are inadvertent, 
for example pavement failures, or due to pavement design, such as expansion 
joints, railway, and tramway crossings, manholes or speed bumps [7]. 


Pavement failures can be summarized as cracks, surface deformations, and 
surface defects [8]. Surface deformations mainly include ruts and shoving, 
surface defects represent bleeding, raveling, and potholes. The repair proce- 
dure is to determine the type of pavement failure, determine the cause, fix the 
cause and to perform the repair [8]. 


1.2 Road condition 


1.2.1 Causes for road defects 


The main criteria to be met by the road infrastructure are [9] 


e evenness, tire road friction, and carrying capacity, 
e brightness and reflectivity, 

e drainage of water, 

e low noise levels. 


With a systematic maintenance, the life span of asphalt layers is 10 to 15 years 
and 15 to 20 years for binding layers. Concrete surfaces can last up to 30 years 
and base layers 40 to 50 years [10]. 


The life span of a road is limited by either structural aspects and the aging 
of the construction materials or by outside influences, such as the weather 
condition and traffic load. For example, binders naturally lose their binding 
and restoring force over time. Furthermore, the wear of the road occurs due to 
tire road contact. 


One of the most important external factors influencing the state of the roads 
is heavy-load vehicle traffic and the axle load in particular. The axle load is 
the proportion of the total weight attributed to one axle of a vehicle. The total 
weight of the vehicle consists of its own and loaded weight. A long-term test 
showed that a linear increase in axle load leads to a damage of the road in 
fourth power [11]. Since 2002 the traffic volume of commercial road haulage 
has increased by more than 50 % in Germany, from 1,454 to 2,245 million 
tonnes annually [3]. 


In addition to traffic loads, the season is the main factor, which influences 
the road condition. Particularly long-lasting heat periods lead to damages 
on asphalt and concrete roads. For example, the concrete plate stretches due 
to solar radiation and reduces the transverse joints. The plates can break if 
there is not enough space and cause a ripple, which is referred to as blow 
up [12]. Asphalt has relaxation capacity because of bitumen and expands with 
constant stress and increasing temperature and softer bitumen. The results are 
deformations and cracks on the surface [12]. 


The condition of the road infrastructure can significantly deteriorate after a 
period of very cold weather. An aged or cracked road surface allows the entry 
of water into pores and cracks. The water freezes by frost into ice crystals, 
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which enlarge the road damages. Repeated melting and solidification and 
traffic loads result in the formation of potholes [12]. 


1.2.2 Effects of road defects 


Many studies have investigated the influences of bad road conditions [13-15], 
the increase of tire road noise was proven by the experiments of the author 
[16]. The influences can be summarized as 


e increase of tire road noise, 

e decrease of driving comfort, 

e decrease of driving safety, and 
e increase of rolling resistance. 


These influences result in even more undesired effects. For example, a high 
level of tire road noise leads to general annoyance, sleep disturbances, or 
speech interference [17, 18]. Furthermore, it increases the costs for noise 
abatement measures, such as sound-absorbing barriers [19]. Tire road noise 
also yields to a decrease in value of houses and land close to noisy road 
transport infrastructure [20-22]. 


Road defects not only decrease the driving comfort and affect the behavior of 
the vehicle [23] but can also cause damage to the vehicle. A deep pothole 
might deform the wheel rim and constant higher load impacts the wear of the 
rubber mounts. Particularly bicycle riders are vulnerable from rough surfaces, 
which increase the danger of fall or injury. 


An increase of rolling resistance leads to higher carbon dioxide emission and 
vehicle operational costs, such as the fuel consumption, and lower range of 
electric vehicles [24]. Moreover, if the road condition exceeds a critical value 
due to delayed detection of road defects, the erosion of the road substance 
accelerates. Subsequently, the road needs a complete renewal and the costs 
increase over the life cycle of the road [25]. A renewal often yields to a complete 
roadblock and to traffic jams. Overall, repairs result in high economic costs 
due to the time spent in traffic jams and diversions. 


1.3 State-of-the-art of road condition monitoring 


1.3 State-of-the-art of road condition monitoring 


The current applied processes for road condition monitoring were analyzed 
based on manuals from transportation departments around the world, e.g. [26— 
32]. The condition monitoring process is often based on several steps, which 
involve the data collection, defect identification, defect assessment, and the 
calculation of a road condition index [4]. However, the execution and details 
of each step differ from country to country. 


The road infrastructure is often scaled in segments of a specific length, which 
usually ranges from 10 to 100 meters among the countries. 


The data collection is mainly performed either manually by accredited survey- 
ors, who either walk or drive along the roadway, or automatically by specialized 
vehicles, which are equipped with various sensors. Latter vehicles create high 
costs of purchase and maintenance and only few countries use such vehi- 
cles [33]. In Germany for example, the Federal Highway Research Institute 
performs measurement campaigns with these specialized vehicles on interur- 
ban roads in fixed intervals of multiple years. However, their use is limited 
to the primary road network and a manual data acquisition is performed in 
communes. 


The types of defects under consideration of most of the countries are mainly 
different types of cracks, rutting, shoving, distorting, potholes, raveling, and 
bleeding. The defects are assessed manually on site, are based on pictures and 
data from measurement vehicles or through an index calculated on the basis 
of the severity and type of defect. Finally, an overall road condition index is 
calculated based on the defects and their severity or density, which differs in 
weights and calculation across the transportation departments. 


The different equations for this index from various states in the USA are 
evaluated in [34]. Based on the assessment of the quality of the road segments, 
a priority rating for repair is assigned by the inspectors. 


In many countries the international roughness index (IRI), a slope statistic 
in m/km of road segments is calculated with road profilometers. There are 
different International Quality Levels IQL-1 up to IQL-3), but even the most 
accurate IQL-1 systems typically report the roughness at 10 to 20 m intervals, 
while IQL-3 systems only report at intervals greater than 100 m. Hence, for the 
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purpose of identifying defects the IRI is of limited use, as further information 
would have to be collected in some other way, before specific reconstruction 
planning can take place. 


Interviews with experts were conducted to collect qualitative data and infor- 
mation about the current status and development of road condition monitoring 
that could not be found in the literature [198]. The focus of the interviews was 
on the area of Karlsruhe, Germany. An expert should have special knowledge 
which, in contrast to general knowledge, involves complex, integrated knowl- 
edge and is also constitutively related to the pursuit of a profession [35]. Three 
experts with the described special knowledge were selected!. 


They stated, that the costs for a specialised measurement vehicle are approx- 
imately 120 Euro per kilometre [198]. A visual road condition assessment is 
conducted from inspectors every two years in Karlsruhe and the road segments 
are classified into five groups of quality [198]. However, the approach in small 
municipalities is different. Thus there are communes, which do not measure 
their state of the road at all [198]. There is a clear need for a continuous mon- 
itoring system, since environmental influences can cause short-term changes 
ofthe road state, especially after a strong winter or a hot summer [198]. A real 
time road condition monitoring system would be a huge step forward [198]. 


1.4 Survey of related research 


1.4.1 Overview 


Researchers have addressed the problem of a manual, expensive, and discon- 
tinuous process to monitor the road infrastructure and present concepts and 
methods for a mobile data acquisition and automated processing to estimate 
the road roughness or to detect road hazards. Figure 1.1 gives an overview and 
the structure of relevant research for this chapter. 


! The experts are Univ.-Prof. Dr. h.c. Ralf Roos from Karlsruhe Instiute of Technology, Prof. 
Dr.-Ing. Markus Stockner from Karlsruhe University of Applied Sciences, and Manfred Geiger 
from the civil engineering department of the city of Karlsruhe. 
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The measurement sensors, which have been applied to monitor the road infra- 
structure are mainly 


e road profilometers, 
e cameras, 

e acoustic sensors, 

e inertial sensors. 


{ automated and mobile RCM (> 100)) 


sensor 
RP (~10) [€ (>50) ) AS (~5) IS (~30) 
method 
(other) (other (<10)) (ML(>40)) (other) (PM(~15)) (ML (~15)) 
output 
P (attributes } (attributes ) (RI) (RI) (RI) | attributes ) 


RCM: road condition monitoring 

RP: road profilometers, C: camera, AS: acoustic sensor, IS: inertial sensor 
ML: machine learning, PM: physical model 

RI: roughness index 


Figure 1.1: Automated and mobile road condition monitoring (RCM) research structure and the 
amount of research papers in parentheses. The first level indicates the sensor, the 
second level the method and the third level the output. The output is mainly either 
road attributes, e.g. defects, or a roughness index (RI), such as the IRI. Nodes anno- 
tated with ’other’ for the data processing method level means, that the method was 
not clearly specified in the paper or the method can not be assigned to the classes PM 
or ML. 


The numbers of research papers indicate, that there has been a lot of research 
with cameras and mainly machine learning (ML) methods were applied. There 
only exists few studies which investigate acoustic sensors to monitor the road 
infrastructure. More publications can be found for road profilometers and 
inertial sensors. 
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With latter sensors, physical model or heuristic models with ML algorithms 
can be applied to estimate the road condition. The majority of studies, which 
use inertial sensors and ML, output a roughness index and only few predict 
defects. The literature applying inertial sensors and ML is reviewed in more 
detail in subsection 1.4.5 with focuses on specific subjects, such as labeling of 
training data or feature selection. 


The level of detail of the output for an automated road condition monitoring 
can be distinguished into in three categories [36]. The categories are 


e presence: methods, which are only able to determine whether a defect 
does or does not exist in the given data, 

e detection: methods, which are capable of identifying the exact defect 
position within the given data, 

e measurement: methods, which are able to provide the spatial measure- 
ments of a defect, such as the width, length, and depth. 


Table 1.1: Evaluation of sensors based on previous literature. 


Sensor Defect Level of detail Cost Effort 
Profilometer all defects measurement high high 
Camera 2D surface defects detection medium medium 
Camera 3D surface defects measurement medium medium 
Acoustic sensor surface type - low low 
Inertial sensor elevation defects presence low low 


Table 1.1 summarizes the sensors, considered defects, level of detail, invest- 
ment, and operating costs and the effort to install the sensor and operate the 
measurement system, based on the reviewed literature. 


1.4.2 Application of road profilometers for road condition 
monitoring 


Mobile road profilometers can be distinguished in tactile and laser profilome- 
ters. The longitudinal profile analyzer (LPA) serves as an example for mobile 
tactile profilometers [37]. The measurement system consists of one or two 
trailers with one suspension and wheel each, which is in continuous contact 
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with the road. A movable transverse beam transmits the vertical movement of 
the wheels to the data acquisition unit. The road profile from this system is 
used as reference for other road condition methods, for example in [37, 38]. 


3D laser road profiling systems were investigated in [39-41]. They are able 
to identify most types of road defects and vehicles with such systems can 
travel with speeds of up to 100 km/h [42]. Data processing techniques are also 
of high standard and researchers were able to drastically reduce computation 
times for 3D dynamic image optimization in recent years [43]. However, the 
costs of equipping one vehicle with laser line profiling technology are quoted 
up to 220,000 USD [44] and they have to be handled by trained operators to 
ensure proper functionality [45]. Furthermore, objects on the road, such as 
sand, gravel, salt or snow negatively influence the results and lead the system 
to misidentify road defects. 


1.4.3 Application of cameras for road condition monitoring 


There has been a lot of research in using cameras for an automated road 
condition monitoring. A recent literature review to camera road condition 
monitoring gives an overview of previous research focused on camera sys- 
tems [36]. 


Most of the proposed methods operate on 2D images, e.g. [5, 46-60], while 
there are few 3D reconstruction methods that utilize the notion of stereo- 
vision for detecting road defects or road markings [61-67]. 3D reconstruction 
methods offer the possibility to reconstruct the surface using images from 
various video cameras. A stereo camera is utilized in the Magic Body Control 
from Mercedes-Benz, first introduced in 2013. The camera is located behind 
the windscreen just in front of the rear view mirror and scans the road surface 
ahead to detect changes in the road elevation. With this information, the 
suspension is actively adjusted to reduce vehicle body movements [68]. 


Research with cameras was mainly performed to detect road hazards but not to 
estimate the road roughness. Table 1.2 summarizes the most cited papers with 
the applied method, type of object recognition and level of detail, focusing on 
road condition monitoring with camera and an automated data processing. 
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Table 1.2: Overview of literature employing camera based methods. 


Method Objects Level of detail Literature 
3D reconstruction potholes measurement [60] 
ANN cracks presence [46-48] 
ANN cracks detection [49, 50, 52, 53] 
STF cracks, patches, potholes detection [36] 
SVM patches, potholes detection [59] 


Machine learning multiclassifier algorithms nowadays are the most common 
methods for object detection in images. They allow simultaneous segmentation 
and recognition of several objects in images [69-71]. Since the defects can be 
seen in the image and labeled by an annotator, mainly supervised learning with 
either artificial neural networks (ANNs) or support vector machines (SVMs) 
were applied. A semantic texton forest (STF) was applied in [36], as it can use 
various features for segmentation, namely texture, layout and context. 


The results of the reviewed studies show, that cameras in combination with 
machine learning methods are able to precisely detect and mark defects within 
images. They are inexpensive and modern vehicles with special packages are 
equipped with cameras ex works so that comprehensive coverage of the road 
network can be achieved. 


However, cameras have general features that present definite disadvantages. 
Firstly, their use is limited to decent weather conditions, since rain, fog, mist, 
or snow can make capturing images of sufficient quality impossible. A second 
limitation is given by the speed at which vehicles can travel without the vi- 
bration and suspension movements affecting the sharpness of the image. The 
authors of a recent paper conceded that they had to drive at speeds between 10 
and 15 km/h to keep unexpected vibrations of their vehicle at a minimum to not 
affect the quality of their data [36]. One solution is to use cameras that possess 
high enough resolution, frame rates, and image stabilizing to warrant accurate 
measurement results at higher speeds. However this leads to increasing costs 
for hardware and reveals the trade-off between the quality of the picture and 
the corresponding amount of data storage required. 


Throughout the corresponding literature, a general problem with ML is the al- 
gorithm training phase. Even the most recent papers have to rely on supervised 
training and manual labeling of training data. This is quite time-inefficient and 
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thus limits the amount of valuable training data available. Hence ways to label 
data more efficiently, increasing the amount of training data, and thus improv- 
ing the accuracy of the algorithms or the application of unsupervised learning 
should be of high interest for future research in this field. 


1.4.4 Application of acoustic sensors for road condition 
monitoring 


One limitation of indirect methods to estimate the road roughness based on 
accelerometer measurements is the limitation in frequency range. High- 
frequency excitations from the road surface, for example the macrotexture, 
are filtered by the tire and suspension, which act as a low-pass filter [45]. 
However, passing from smooth to coarse asphalt results in audible changes in 
level and frequency distribution of sound, which can be captured by micro- 
phones attached to the vehicle [45]. Results from previous research indicate 
that methods based on sound have to compensate the current vehicle velocity. 
Further limitation are the effect of snow and water on the road on the sound 
measurements. 


Microphones in or outside the vehicle body have the disadvantage that noise 
from the environment, such as other vehicles and reflections, influence the 
data. An alternative method to analyze the sound pressure in the tire cavity 
was developed in [72]. The road surface oscillates the tire and the vibrations 
generate sound waves in the tire cavity, which is a reverberation room and 
insulated from the environment. 


The measurements of the studies investigating tire road noise show a good 
reproducibility. Moreover, they demonstrate the potential to distinguish be- 
tween road surfaces with different characteristics, especially various macro- 
textures [73-75]. However, an acoustic sensor in the tire cavity has only been 
used under specific driving conditions and no automated data processing has 
been developed. 
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1.4.5 Application of inertial sensors for road condition 
monitoring 


Inertial sensors and physical models 


In computer simulation, researchers have investigated the possibility to estimate 
the road profile based on physical vehicle models but have not considered 
to detect road defects or other road features. The estimation in computer 
simulation is mainly based on a half-car models with inertial sensor data from 
the sprung and unsprung masses of the vehicle model [76, 77]. 


The inversion from the vehicle model, where usually the road profile serves as 
input and the accelerations as output, is acommon method to derive the road 
profile [199]. The applied methods with real data are mainly based on sliding 
mode observers [78, 79] or filters [38, 45, 80-83], such as Kalman filters. The 
basis of these methods with real data is mainly a quarter car model. 


Overall, the results of these reviewed studies show that the methods give a good 
estimation of the road profile. However, the accuracy of the estimation is prone 
to changes in velocity. Moreover, the transportation departments concentrate 
on detecting type of defects, which requires an additional step in the data 
processing of the methods employing inertial sensors and physical models. In 
contrast, methods using ML can directly output the type of defect. 


Inertial sensors and heuristic models 


Machine learning approaches for road condition estimation have been mainly 
applied for real driving rather than in computer simulation. The methods can 
be distinguished in their output, which is either a roughness index or defects. 
Table 1.3 gives an overview of recent literature employing inertial sensors and 
a heuristic model. 


In computer simulation, studies applied ANNs to estimate the road profile with 
data obtained from vehicle dynamics models [84—88], mainly half-car models. 


A method to detect potholes based on acceleration sensors and global position 
system (GPS) for localization is presented in [89]. Seven taxis were equipped 
with the sensor system, which managed to cover 2,492 distinct kilometers 
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Table 1.3: Overview of literature employing inertial sensors and heuristic models. 


Method Output Level of detail Literature 
ANN roughness [84-88] 
filters potholes presence [44, 89] 

RF roughness [88, 90, 91] 
SVM roughness [88, 92] 
SVM elevation defects presence [93-96] 


during ten days. Various signal processing filters were applied for data analysis 
a misclassification rate of 0.2 % was reported. Research in this field with 
similar filters was extended by [44], who equipped 100 taxis in the Shenzhen 
urban area with low-end accelerometers and GPS devices to obtain vibration 
pattern, location, and vehicle velocity. The advantages of machine learning 
approaches over physical models was pointed out by [85]: 


e The use of neural networks does not require excessive system character- 
ization. In fact, "all it requires is the road profile data representative of 
various degrees of road roughness as network targets, and accelerations 
measured on the vehicle system as network inputs.". 

e They require relatively fewer analytical skills to create the network than 
the physical model. The development of physical models still demands 
high technical expertise even for computer simulation based approaches 
and it requires the calculation of inverse models to determine road pro- 
files from vehicle acceleration measurements. 


A comparison of different machine learning models, namely ANN, SVM, and 
random forest (RF) was performed by [88]. 


Research in road condition monitoring with inertial sensor data is becom- 
ing increasingly popular. The main advantages are that modern vehicles are 
equipped with inertial sensors and in contrast to camera systems, this method 
is not limited to decent weather and daytime light conditions. 


The limitations are that each vehicle has to be calibrated to compare the 
results [80]. Moreover, tire road contact on the area to be measured has to be 
ensured. In contrast, cameras can detect defects between or next to the two 
vehicle lanes. However, a large number of surveying vehicles with inertial 
sensors might diminish the disadvantage of the limited measured area and 
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collect data of the most interesting area of the road. The level of detail is 
limited and the determination of the attributes of road defects has not been 
presented. 


Another issue that arises is the labeling of training data acquired with inertial 
sensor. In contrast, labeling methods for camera data presented in research are 
mostly coherent and automated. The presented annotating methods of inertial 
sensor data range from describing the road features into a microphone while 
acquiring training data, or trying to match road images with the corresponding 
inertial sensor data, which is performed offline. 


Review of comprehensiveness and generalization Comprehensiveness 
and generalization of a method are of utmost importance, as the aim of an 
automated road condition monitoring is to be able to work in all driving and 
environment conditions. To test the generalization, experiments have to be 
comprehensive in the variation of vehicles they use, road segments to be mea- 
sured and the amount of kilometers of road network covered. Many papers only 
use computer simulated data to test the accuracy of their algorithm and others 
use only specific vehicles on fixed test tracks, a few kilometers of selected road 
stretches, or consider only few and specific outputs. 


For example, Eriksson, Girod, Hull, et al. [89] equipped seven identical taxis, 
which covered the road infrastructure in Boston over a ten day period. Menant, 
Martin, Meignen, et al. [97] used 12 similar probe vehicles on a 1000 km road 
network in the north of France to measure the IRI and Chen, Lu, Tan, et al. [44] 
deployed a road surface monitoring system on 100 taxis in Shenzhen but did 
not provide information of the type of vehicle. Seraj, Zwaag, Dilo, et al. [96] 
used 5 different vehicle types, including hatchbacks, superminis, and an SUV 
and covered a total of 45.9 distinct kilometers in the testing phase. However, 
they did not report if and how they combined the measurement data from the 
different vehicles or if they trained each vehicle separately. 


Review of methods for feature selection Supervised ML approaches 
mainly classify based on features rather than time series. Nitsche, Stiitz, 
Kammer, er al. [88] state, that "In order to use the proposed machine learning 
models effectively, one of the most important steps is to extract expressive 
features from the collected data". 
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Only few papers provide insight on their feature selection. Tai, Chan, and Hsu 
[93] modify some of the 59 roughness parameters for measuring road surfaces 
presented in [98]. Finally, they use a total of six distinct features extracted 
from their input data, which includes the acceleration values in three axes, 
instantaneous speed, and position. The features are the mean speed and the 
mean, range, standard deviation, maximum, and minimum of the acceleration 
for each axis. 


Perttunen, Mazhelis, Cong, er al. [94] use sliding windows of 2 s length with 
a 0.5 s slide to extract features from the acceleration signals. They utilize the 
backwards feature selection algorithm of PRTools [99] to select optimal speed 
scaled and non-speed scaled feature sets. 


Nitsche, Stütz, Kammer, er al. [88] used a total of 35 features as input for the 
models, gained from the longitudinal, lateral and vertical acceleration as well 
as the angular speed of each wheel. Among the selected features were the 
mean, standard deviation, range, and the mean of the short time energy as well 
as the power spectral densities of the accelerations, and angular speed of both 
rear wheels, respectively. 


To prepare data for the classification phase Seraj, Zwaag, Dilo, et al. [96] 
compute features from time domain, transformation in frequency domain and 
wavelet decomposition for a total of 18 different features. 


Laubis, Simko, and Schuller [90] and Laubis, Simko, Schuller, er al. [91] use 
the same features based on GPS, 3-axis accelerometer, and 3-axis gyroscope 
measurements for both papers to estimate road roughness. They perform a 
continuous wavelet transformation to analyze the frequency content of the 
acceleration and gyroscope features. Other aggregation functions used are the 
mean, range from minimum to maximum, standard deviation, variance, and 
root mean square (RMS). Overall, most studies use 35 or even more features 
for finally estimating the IRI or a similar roughness index. 


Review of performance and stability of methods Amongst the different 
machine learning methods, support vector machines (SVMs) have shown to 
be most popular recently and outperform other methods, e.g. random forest 
(RF) and artificial neural network (ANN) [88]. For example, Nitsche, Stiitz, 
Kammer, et al. [88] state that "Choosing a set of representative features is 
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important for each model, but the RF and SVM are found to be more stable 
than the ANN with respect to the quantity of features. For example, a drastic 
reduction of features (from 35 to as low as 5) does not significantly alter the 
results for the former two, while a neural network is not able to capture the 
structure of the data efficiently in this case.". 


Review of methods for labeling training data An important step in ensuring 
the accuracy of any selected machine learning algorithm is providing sufficient 
amounts of correctly labeled training data. Without a large enough pool of 
training data to learn from, the performance of any algorithm is severely 
hindered. 


While some papers do not describe their labeling technique for real tests and 
others rely entirely on computer simulations, where the data can be automat- 
ically annotated, a couple of methods for labeling real world data have been 
used with varying success. 


Eriksson, Girod, Hull, et al. [89] have used a computer in the vehicle and 
a trained annotator, who sits in the passenger seat of the vehicle, presses 
keyboard keys corresponding to predefined defect classes when they occur. 
However, they also used a set of loosely labeled data, where the types and 
rough frequency is known but not the number and location of the defects. 


Tai, Chan, and Hsu [93] use a voice recorder to label the training data during 
collection with motorcycles in Taiwan. They chose to do this to minimize 
the required interaction of the motorcycle driver, but had to concede high 
inaccuracy and subsequently had to develop a very elaborate relabeling process 
to make their training data usable for the algorithm they used. 


Perttunen, Mazhelis, Cong, et al. [94] covered 25 km of road in one drive 
capturing video data next to the accelerometer and gyroscope readings. They 
preprocessed global position system (GPS) and acceleration signals and plot- 
ted a spectrogram of the sum of the band energies of accelerometer signals, 
superimposing speed in the same figure. 


Two independent annotators then produced two label files by analyzing the 
video data obtained during the drive. This method of training data labeling is 
rather complicated and even the authors concede that labeling data from video 
is time consuming and error prone work. 
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In the work from Seraj, Zwaag, Dilo, er al. [96], labeling was performed by one 
person sitting in the passenger seat using the microphone and the camera of the 
smart phone. They took various drives over the same road segment at different 
speeds and hitting the defect in different angles, with the annotator loudly 
mentioning the type of defect as the vehicle approached it. A software was 
used to transcript all voice labels with accuracy in the order of milliseconds. 


Problems arose when synchronizing the subtitle file with the smartphone data 
timestamp and lags between the labeled segments of data and the actual defect 
was noticed. As a result almost all labels had to be manually corrected. 


Review of output metric As shown in Table 1.3, the most common output 
metric is the road roughness, especially the international roughness index 
(IRI). However, as mentioned in Section 1.3 the IRI can only give a generally 
assessment of the quality of a road segment but the knowledge of the severity 
and location of specific defects, such as potholes, is more important for a road 
condition monitoring. 


Few publications distinguish between different types of defects and only con- 
sider three or less classes. Furthermore, no study investigates to differentiate 
between different types of materials, such as concrete or asphalt. 


1.5 Open questions 


The presented literature shows, that road profilometers are too expensive to 
equip various vehicles with and usually have to be attached outside the vehicle. 
Hereby, they are prone to environmental disturbances, such as snow or water. 


Camera systems for road condition monitoring have been extensively studied. 
Further improvements are to precisely measure the attributes of defects and to 
estimate the road profile. With improvements of the hardware, also this task 
will be solved soon. Furthermore, fundamental research in unsupervised and 
reinforcement learning in computer vision might replace the time consuming 
labeling while using supervised learning. 


Acoustic sensors are rarely investigated in terms of comprehensive and auto- 
mated road condition monitoring. However, they seem to be a good option 


17 


1 Introduction 


to monitor road infrastructure and even to estimate the macrotexture of the 
road profile. Previous literature has used acoustic sensors to measure the tire 
road noise, which is also dependent on the macrotexture. An acoustic sensor 
outside of the vehicle has many disadvantages, mainly the negative influ- 
ence from background noise, reflections, among others. However, a dynamic 
pressure sensor in the tire cavity is robust against disturbing noises. So far, 
research investigated the tire cavity sound on test benches or test vehicles under 
controlled conditions. Moreover, the focus was on the correlation with con- 
ventional tire road noise measurement procedures, such as the close-proximity 
method. There is no concept or method to comprehensively and automatically 
apply a tire cavity sound measurement system for road condition monitoring. 
This implies the development of a measuring device, which is inexpensive and 
can be installed on vehicles without a special permit. Furthermore, a method is 
needed, which analyzes the data from the tire cavity sound sensor in all driving 
and environment conditions. 


Inertial sensors for road condition monitoring compensate for various disad- 
vantages of camera systems. They are robust against the environment condi- 
tions and measure the ride comfort, which is mainly dependent on the road 
roughness. 


Previous presented methods to estimate the road roughness, e.g. the IRI, 
have several disadvantages. Physical models use special vehicles with the 
knowledge of the suspension characteristics to acquire the vehicle dynamic 
data. The road roughness is then derived with an inverse vehicle model. 
This method is prone to a change of the vehicle vibration behavior and needs 
additional sensors at the unsprung masses besides an inertial sensor in the 
vehicle body. Recent literature presents a machine learning (ML) approach 
with one sensor in the vehicle body to train vehicles with ground truth data 
acquired with profilometers. However, this method involves a costly training 
process and is mostly limited to road surfaces, which have been trained. A 
method to estimate the road roughness based on a vehicle inertial sensor, which 
is not limited to specialized vehicles or does not involve a training process, 
needs to be investigated. 


The research on road object detection with inertial sensors has still room 
for improvements. Studies using computer simulations mainly focus on the 
estimate of road profiles and do not classify defects, which is an important input 
for road condition monitoring. Furthermore, ML methods, which are applied 
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for real test drives should also be tested and verified in computer simulations. 
A new simulation approach taking these points into account, might be the basis 
for further research on this topic under controlled conditions to analyze the 
influence of specific parameters or vehicle behavior on the performance of ML 
methods. 


For real drive measurements, the research gap involves the acquisition of data, 
especially the process of labeling trainings data. Moreover, a method to inves- 
tigate and evaluate the features to enable a robust and computational efficient 
classification has to be introduced. Previous literature has used many features, 
without giving more insight why they were selected. Too many features in- 
creases the risk of over-fitting and decreases the generalization ability. The 
classification methods so far have been trained and tested mainly under limited 
conditions, e.g. specific velocities, road conditions, road areas, specific vehi- 
cles. Furthermore, a method to combine the output from several vehicles has 
not been presented. 


1.6 Aim of this work and research objectives 


The aim of this work is to induce methods to estimate the road infrastructure 
condition with vehicle based sensors and models for an efficient and automatic 
data processing. It presents the development of a novel measurement system for 
the acquisition of both tire cavity sound and inertial sensor data of the vehicle 
body. Furthermore, a novel simulation approach is developed, which enables to 
test ML methods and vehicle dynamics models with varying parameters. The 
first research question is addressed to the inertial sensor: How can we process 
the inertial sensor data in the vehicle body to provide a road roughness index 
without training and special vehicles? 


The second research question takes the acoustic sensor in the vehicle tire into 
account: How can we estimate the road roughness with tire cavity sound data 
under various driving and environment conditions from various vehicles? 


Besides estimating road roughness, the further research question arise: How 
can we process inertial sensor data from the vehicle body to detect road 
attributes, such as potholes? 
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Lastly, a research question is: How can we combine the output from various 
vehicles to improve the prediction accuracy? 


To address these research questions this work aims to 
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develop and verify a measurement system for a comprehensive and 
automatic road condition monitoring with an inertial and acoustic sensor 
attached to vehicles 

develop methods to automatically record adequate data sets 

develop a novel simulation approach to investigate the impact of parame- 
ter variation of road attributes, e.g. height of potholes, and of the vehicle 
dynamics model, such as suspension stiffness. 

design a processing chain to evaluate road data based on measurements 
of inertial and acoustic sensor 

develop methods to combine the output of various vehicles 

integrate the developed algorithms into a graphic user interface for eval- 
uation of data sets by non-experts. 


2 Fundamentals 


Outline: This chapter describes and explains existing methods, which are 
necessary for the development of the new concept and fundamental for the de- 
velopment of new methods for road condition monitoring with vehicle based 
sensors in this work. Section 2.1 defines the coordinate axis and systems of 
the vehicle. Section 2.2 describes the vibration of the vehicle and the tires and 
is important to comprehend the selection of vehicle sensors for this work and 
the derivation of vehicle models. On-board vehicles sensors are discussed in 
Section 2.3. Vehicle and road models, which are used for the novel simula- 
tion approach, are described in Section 2.4 and data mining methods for the 
developed data processing in Section 2.5. Section 2.6 describes performance 
indices for the validation of the developed measuring devices in this work. 


2.1 Coordinate systems 


According to [100], an axis system has three mutually orthogonal directions, to 
which the X, Y and Z axis are assigned. The standard is based on a right-hand 
system. A coordinate system consists of an axis system and an origin. The 
earth-fixed axis system Xz, Yg, Zg is connected to the location and the plane 
spanned by Xg and Yr is perpendicularly to the direction of gravity g. The 
road plane Xr, Yr, representative for the road surface, is within the tire contact 
area. In the case of an uneven road there is a different road plane for each 
tire contact area with a specific road profile. The vehicle coordinate system 
is based on the vehicle axis system and the origin lies in the vehicle reference 
point, as exemplarily shown in Figure 2.1. The vehicle reference point is fixed 
in the frame work of the sprung mass of the vehicle. In the vehicle axis system, 
the Xy-axis is directed forward and parallel to the vehicle longitudinal center 
plane, if the vehicle is at rest. The Yy-axis is perpendicular to the vehicle 
longitudinal center line and points to the left, the Zy axis points upwards. The 
vehicle reference point can be set at various points whereas the usual place is 


21 


2 Fundamentals 


Figure 2.1: Vehicle with three translational and three rotational degrees of freedom. The ref- 
erence point of the vehicle axis system Xy, Yv, Zy with the angles wy, dv, and 
Oy is placed in the center of gravity of the vehicle (CGy ). One of four wheel axis 
systems Xw 4, Yw 4, Zw 4 is defined with the angles Ow 4, Yw 4, YW 4- 


the center of gravity of the sprung mass (CGy). The longitudinal acceleration 
ax is the component of the vehicle acceleration in Xy direction, the lateral 
acceleration a, in Yy direction and the a, in Zy direction. The vehicle velocity 
vy is the velocity of the vehicle reference point, with which the vehicle moves 
relative to the earth-fixed coordinate system. The yaw angle Wy is around the 
Zy axis, the pitch angle 0, around the Yy axis and the roll angle dy around the 
Xy axis. The pitch and roll angle is not measured relative to the road surface 
but to the earth-fixed coordinate system. Accordingly, a vehicle in stand on 
an inclined road surface also has a non-zero pitch angle. The roll velocity wx 
is the x-component of the vehicle angular velocity w, the pitch velocity wy 
represents the y-component and the yaw velocity wz the z-component. 


The wheel axis system Xw, Yw, Zw for each of the four wheels act in the 
wheel center of gravity CGw and they are defined according to the vehicle 
axis system. The accelerations are denoted by awx, awy, and aw, and the 
angles are defined as the wheel rotation angle 8w, wheel position angle ww, 
and wheel camber angle yw. 
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2.2 Vehicle vibration 


2.2.1 Overview 


Resulting forces from road irregularities induce an excitation of the vehicle 
oscillation system [101]. Since road irregularities occur as excitation with 
different amplitude and wavelength at irregular intervals, the road can be 
described as a stochastic vehicle excitation. Aside the road surface, dynamic 
roll and pitch movements during longitudinal and lateral dynamic maneuvers 
as well as internal excitations from power train and wheel generate forces [102]. 


Furthermore, environmental influences can have an impact on the vibration 
behavior of the vehicle. For example, the eambient temperature influences 
the viscoelasticity and rubber friction [103] and snow or water on the road 
influences the tire road contact. Moreover, the aging and chemical process of 
the materials of the vehicle change the vibration behavior [104, 105]. 


2.2.2 Tire road contact 


The footprint of the tires is the only direct contact of a motor vehicle to 
the roadway. They are responsible to carry the vehicle load, to ensure the 
transmission of force between the vehicle and the road and to compensate 
road irregularities. The two major physical effects responsible for tire road 
friction are adhesive friction and friction from hysteresis due to the viscoelastic 
properties of the tire [102]. In addition, tire characteristics should ensure safety 
in different weather conditions and driving situations, achieve a high degree of 
economic efficiency due to a low rolling resistance and long service life and be 
mechanically and acoustically comfortable with regard to noise, vibration, and 
harshness (NVH). In order to meet these requirements, the tire can be optimized 
through the tread pattern design, rubber compound, contour, and construction, 
and taking into account possible interactions of the components [106]. 


The tire belt behaves as a membrane and vibrates due to the excitation from the 
road surface. Pavement defects, pavement facilities, such as bridge joints, and 
rough road surfaces cause the tire profile to vibrate and subsequently the tire 
belt and the tire sidewall [7]. These vibrations lead to radiation of sound waves 
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in the tire cavity and to the environment. In addition, the sound is transmitted 
as structure-borne noise over the rim into the interior of the vehicle. The sound 
emission to the environment is mainly responsible for the tire road sound, 
which has long been a problem for industry and research to reduce [106]. 


2.2.3 Vehicle body vibration 


The mass, stiffness and damping distribution in the structure and the vibration 
excitation of the structure is responsible for vibration behavior of a vehi- 
cle [107]. The vehicle is a multibody vertical dynamic system with various 
degrees of freedoms (DOFs). The objectives of safe vertical dynamic charac- 
teristics are mainly low vehicle body accelerations, low roll and pitch motions, 
low dynamic wheel load fluctuations as well as a load independent vehicle 
vibration behavior [108]. The vertical forces essentially consist of spring 
and damper forces, which ensure the support of the vehicle body relative to 
the suspension system and limit the movements of the vehicle relative to the 
roadway [102]. Today, usually coil and air springs are installed in vehicles. 


Table 2.1 shows several vibration phenomena that are caused by the road 
surface [102]. Other vibration phenomena, such as bucking, are caused by 
imbalance or power train. 


Table 2.1: Influences of road surface on ride comfort [102]. 


Vibration phenomena Frequency (Hz) 
from to 
Vehicle body vibrations 0.5 5 
Freeway-Hop 2 5 
Shake 7 15 
Bouncing 7 25 
Axle vibration 10 15 
Buz 30 70 
Axle roughness 30 80 
Roll motion 30 300 
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2.3 On-board vehicle sensors 


Nowadays, vehicles are equipped and delivered ex factory with various sen- 
sors [109]. Electronic control units (ECUs) and sensors exchange data through 
the controller area network (CAN) bus, which allows the micro-controllers 
and other devices to communicate in almost real-time. The purpose of the 
sensors and ECUs are mainly to increase the safety of the vehicle passengers, 
the driving comfort, to improve the in-vehicle entertainment, and to control the 
entire operation of the vehicle, such as engine control, transmission control. 
Almost all vehicles today feature safety systems, such as anti-lock braking 
system (ABS) and electronic stability control (ESC), which require a yaw rate 
sensor, steering angle sensor, longitudinal and lateral accelerometer, and wheel 
speed sensors. More sophisticated automotive systems, such as adaptive cruise 
control (ACC), vehicle dynamics control (VDC), magic body control (MBC), 
need information from additional sensors on-board, such as stereo camera, 
pitch and roll rate sensors, vertical accelerometers, and radar. 


Another evolution of car sensing technology is the vehicle-to-everything 
(V2X) communication, which includes vehicle-to-vehicle (V2V) and vehicle- 
to-infrastructure (V2I) communication among others. It allows applications, 
such as platooning, road works warning, or forward collision warning. 


The availability of sensors on-board greatly depends on the automotive pack- 
ages. Since various sensors are already on-board in modern vehicles to measure 
the operating state of the vehicle as well as its environment, novel applications 
arise [109]. These are, for example weather and environmental sensing to de- 
velop local micro-climate urban emission models [110, 111], driver behavior 
characterization [112], or road safety to identify dangerous road network por- 
tions from ABS or emergency braking or to estimate the road quality, which is 
discussed in this work. 


While the basic function of the systems of different original equipment manu- 
facturers (OEMs) are similar as well as the employed sensors, the exact name of 
the automotive packages and the utilized sensors slightly vary between OEMs. 
For example, the package for an active suspension is called Active Body Con- 
trol by Mercedes Benz. Continuously Controlled Chassis Concept by Volvo 
and Dynamic Chassis Control by Volkswagen adjusts the characteristics of the 
damping system. Furthermore, the details of the sensors are often not publicly 
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available due to the intellectual properties policy of the companies and needs 
to be gathered through various press releases about novel automotive packages. 


2.4 Vertical vehicle dynamics and road models 


2.4.1 Overview of models for vertical vehicle dynamics 


Models provide the possibility to investigate and analyze a specific behavior 
under defined and controlled conditions. The characteristics of multibody 
vertical dynamic systems can be determined by vehicle models, such as a 
quarter car, bicycle or full car model [107]. Thereby, the road profile serves as 
the input for the vehicle model and should be modeled as areal road profile. 


2.4.2 Vehicle models 


The mechanical vibration is a more or less periodic motion of a mass, 
which transforms kinetic energy to potential energy and vice versa. The 
Lagrange method can be applied to find the equations of motion of the vehicle 
model [107]. The Lagrange equation applied for vehicle models is 


u PU I a OE aie i=12,...n (2.1) 
dt Ogi Ogi Ogi 

where K is the kinetic energy, U is the potential energy of system i, and D is 
the dissipation function, which incorporates the damping of the system. F; is 
the force, which is applied on the mass m;, q; are the generalized coordinates, 
q; are the generalized velocities and n the number of DOFs. The finite element 
method as an alternative to model a vehicle can be more accurate but needs 
more computation time. It is mainly applied for specific and complex parts of 
the vehicle, such as the tire. 


A vertical dynamic model of a vehicle mainly consists of masses, springs and 
dampers. Thereby, the mass stores kinetic energy and the spring potential 
energy. The damper serves as the dissipative element. The sprung mass ms 
of the vehicle is the mass, which is supported by the suspension [100]. The 
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suspension consists of springs, shock absorber, control arms, among others. 
However, in vibration models often only springs and dampers are considered. 
The unsprung mass m, is defined as the mass, which is not supported by the 
suspension but by the tire [100]. At a specific working point of the spring, the 
spring stiffness k, is defined as the change in force to deflect the spring divided 
by the change in deflection q. In contrast, the damping force is proportional to 
the velocity g. 


For vehicle models usually a constant spring stiffness and damping coefficient 
is assumed. With these assumptions, the kinetic energy K, potential energy U, 
and Rayleigh dissipation function D are 


=5 > 2 thy, (2.2a) 


i=l j= 
n n 


=> D 2, xjkijx;, (2.2b) 


i=l j= 
n n 


=5 ad ci Xj. (2.2c) 


i=1 j= 


Figure 2.2 shows a quarter car model, which is most employed to investigate the 
vibration behavior of a vehicle [107]. Thereby, the tire stiffness is represented 
by a spring with spring stiffness kz. 
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Figure 2.2: Quarter car model. 
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With (2.2a), (2.2b), and (2.2c) and employing (2.1), we get the differential 
equations of the motion: 


ËsMs = —Cs(Zs — Zu) — Ks (Zs — Zu), 


Zumu = Cs(Zs — Zu) + ks (Zs — Zu) + ki(rR — Zu). 


2.4.3 Road models 


A road profile for a quarter car model or various individual road profiles for 
half car or full car models have to be modeled. Hereby, the road profile 
for simulation is supposed to represent a real road profile. In general, the 
unevenness density of a real road profile decreases with an increase in spatial 
frequency. 


Table 2.2: Degrees of roughness [113]. 


Road Lower Upper 

Class limit limit 

A - 2 

B 2 8 Ex 

C 8 32 g 

D 32 128 6) i 

E 128 512 

F 512 2048 i 
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spatial frequency (cycle/m) 


Figure 2.3: Power spectral density of road pro- 
files. 


The power spectral density (PSD) of a road profile is defined in [113] as 
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where Q = 27 /Ir is the angular spatial frequency and lr the wavelength of the 
unevenness. 


Usually, nı = 2, n = 1.5, Qo = 1 and the degree of roughness Gg(Qo) can be 
obtained from [113] and are shown in Table 2.2 for different road classes. The 
unit of Gg(Qo) is 10”°m? and the unit of Qo is rad/m. 


Figure 2.3 shows the power spectral density for four roads with different values 
for degree of roughness. Hereby, road class F represents a very uneven road 
and A a new asphalt in very high quality. 


A periodic road profile can be generated with a Fourier series, the sum of 
individual sine waves 


N 
2 
rR(x) = X ôi . sin .i-x+e;), 
i=0 


where e; as a random phase angle uniformly distributed between 0 and 27 and 
i the running index. A random profile to fit a PSD described in (2.4) can be 
generated with a setting from the literature, e.g. from [114, 115]. 
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Figure 2.4: Road profile generated with (2.5) with four different values for degrees of roughness. 


With f; = V27 ; AQ - Ga(Q) and with the frequency resolution AQ. 
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N 
re(x) = >) ¥2-AQ- Gali AQ) -sin(2n-i-x-AQ +e;). (2.5) 
1=0 


Figure 2.4 shows road profiles generated with (2.5). The characteristics of the 
PSD of these road profiles are similar to the PSD in Figure 2.3. 


2.4.4 State space 


The differential equations of the motion from the vehicle model with the road 
model as input can be transformed into a state-space representation, which is 
often applied for multiple input multiple output systems. State-space equations 
consists of a set of input, output and state variables. Latter variables present 
the status of the system. The advantage is that 


e the representation is compact, 

e it replaces n order linear differential equations with a first order matrix 
differential equation, 

e directly provides a time-domain solution, 

e is computational efficient. 


For a linear, time-invariant model with n,, inputs, ny outputs, and nx states, the 
state-space model is 


X= Asx + Bru, 


y= Cox + Dru, 


where As is the state matrix with dimension As € R"x””"x and By is the input 
matrix with By € R"**"“, which describes the properties of the system. The 
output is determined by Co as the output matrix with Co € R">”"x and Dr as 
the feedthrough matrix with Dr € R"y*"“. u € R”! is the input vector, x € R”! 
the state vector and y € R”» the output vector. 


nx is the minimum number of state variables needed to describe the system, 
which responses to the initial condition x(0) and a set of inputs u(t) for t € R>? 
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For the state-space model of the quarter car model in Figure 2.2 with the input 
u = rr and state vector and output vector 


0 1 0 0 0 
ks Cs ks £s 0 
As = Ms Ms Ms Ms Br = 
> > 
0 0 0 1 0 
ky Cs kı+ks Cs kt 
Mu Mu Mu Mu Mu 
_ ks cs ks cs 0 
_ m m, m, m _ 
Co=| ko ce _kytke e |? DF ke 
Mu Mu Mu Mu Mu 


2.5 Data Mining 


2.5.1 Overview 


There are mainly three types of machine learning tasks, which are supervised 
learning, unsupervised learning and reinforcement learning [116]. There is a 
lot of research and there are several approaches and algorithms of all the three 
machine learning tasks. A detailed discussion and weighting of all possible 
methods of machine learning and the combination of those is beyond this work. 


However, the main goal of this study is to classify and estimate the road 
condition based on features extracted from sensor data, which are physically 
comprehensible. Furthermore, it should be possible to evaluate the accuracy 
of the classifier. These points lead us to apply a data mining method, which 
is based on supervised learning. The advantages of supervised learning are 
that less training data is needed and the output is comprehensible. The main 
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disadvantage is the costly labeling of data to train and test the models. Lat- 
ter drawback can be narrowed down by using data, which are automatically 
acquired and labeled in a simulation. 


The steps of the methods based on supervised learning, which are applied in 


this work, are shown in Figure 2.5. 
(Sensor) analog data ac- feature 
data quisition extraction 
Y 


RER Be en Tr feature 
N 


p 
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class classification feature 
metric regression aggregation 


© testing ` “< training ` 
Figure 2.5: Overview of methods to predict classes or a metric based on time series. 
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Features are extracted from time series acquired from sensors to reduce the 
dimension. The dimension of the feature vector can be reduced through 
feature selection (subsection 2.5.4) and feature aggregation (subsection 2.5.5). 
The classification (subsection 2.5.2) and regression model (subsection 2.5.3) 
needs a training phase to find a function to separate the classes or to find 
the relationship between the dependent variable and the explanatory variables. 
From testing and training phases, performance measures can be calculated to 
estimate the generalization (subsection 2.5.6). 


The input matrix for the classification or regression problem is matrix X with 
X € R’*", which consists of n various features and h feature vectors x; with 
xi € R”. The input matrix X is defined as 


T 

X11 X12 u Xin X 

T 

X21 X22 vv X2n X) 
=]. u = els 
T 

Xhi Xn2 `“ Xhn xX), 
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and the output vector y with y € R” as 


y=(y1,%:.-,Jn) - 


For regression, y; is the observed value, for classification, y; € K;,j = 1,...,1 
is the class corresponding to x;. 


An example of an input matrix to classify road attributes in this study is shown 
in Table 2.3. 


Table 2.3: Example of an input matrix and output for classification. 


Features 
Road segment o(az) (wx) lwy) +: Class 
(m/s?) (rad/s) (rad/s) 
1 0.7 0.21 0.25 one pothole 
2 0.3 0.11 0.13 --» damaged surface 
3 0.1 0.02 0.05 tee smooth surface 
h 0.4 0.12 0.10 ++» damaged surface 


A road segment annotated with a specific class is described by various features, 
e.g. standard deviation from vertical acceleration, roll rate, and pitch rate. In 
case of a regression problem, the output is a metric instead of a class. 


2.5.2 Classification 


The goal of a classification problem is to find a function fx, which returns 
a class based on features extracted from the sensor data. The classification 
function in this study, estimated with annotated data, is used to predict road 
attributes based on vehicle inertial sensor data (Section 3.1). After a training 
process, the model can be tested for generalization in a testing process and 
afterwords the function can predict classes based on new measurement data. 


A support vector machine (SVM) is a model to find such a function [117]. It 
is known as a top performer and it finds a global optimum, maximizes the gen- 
eralization ability, is robust to outliers, and is geometrically explicable [118]. 
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The disadvantages are, that a SVM needs a training process like all supervised 
learning methods and that it has to be extended for a multiclass problem, since 
it uses a direct decision function [118]. The one-against-one method can be 
used to classify various outputs, which was introduced in [119] and firstly 
applied on SVMs in [120]. 


For a multiclass problem, /(/ — 1)/2 classifiers are constructed to find hyper- 
planes between each class. Each classifier is trained on data from two classes 
p and q, notated as (p,q), which leads to the following classification problem: 


ot, Son Dory + 2 

subject to (WD) da) 4b?) >1-EP, if y;=p 26) 
(wD) A(x) + pPD <-] +P, ifyi=gq 
&P9>0. 


C X; éP- Dis the penalty term with the penalty parameter C and slack vari- 
ables €;. The goal is to find a balance between the training errors and the 
regularization term 5 (wi? -2))? w(?-4), where w is the orthogonal vector of the 
hyperplane to separate the classes. b is the shift of the hyperplane. 


The input x; can be mapped into a higher dimensional space with the feature 
map ¢. The minimization problem (2.6) can be transformed into a dual problem 
and the classification rule for a binary classification problem is then 


Ff (xi) = san((w PT px) + bP). 


The instance is finally classified with a voting strategy, which is called "Max 
Wins’ strategy [121]. The vote for class p is increased by one, if x; is predicted 
as class p. Otherwise one vote is added to class q. Finally the class with the 
most votes determines the instance classification. 
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2.5.3 Regression 


A linear regression function is defined as 
fxi) =x;" p +b, 


where x; is the input, B the regression coefficients and b the error variable. 
The regression coefficients can be estimated by minimizing the distance of the 
ground truth y; and f(x;). Common estimation techniques for linear regression 
are for example linear least squares. 


A version of SVM was introduced in [122] to find such a function with a set of 
training data. For this study, the regression function, estimated with annotated 
data, is applied on acoustic sensor data acquired in the tire cavity to estimate the 
road roughness. The goal of the SVM regression is to minimize the distance € 
of the upper and lower bounds to the true value while being as flat as possible. 
Considering slack variables, & and &*, to allow regression errors with cost C, 
the optimization problem can be formulated as: 


. 1 Ee * 
min 5B B+) Eten 


subject to y;—-(x:TB+b)<e€+ći 
(x,7B+b)-yi <e+& 
épé = 0. 


The e-intensive loss function describes the idea behind the SVM regression: 


fee N if [yi = false 


ly; -f(x;)|-€ otherwise 


Only estimated values with a distance greater e to the observed value are 
penalized. 


Analogous to subsection 2.5.2, the input x; can be nonlinearly transformed 
with the function ¢. The optimization problem can be computationally simpler 
solved in its Lagrange dual formation. 
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2.5.4 Feature selection 


In order to evaluate the features and to select a number ñ of the most important 
ones, the goal is to find a function Q, which describes the feasibility of a 
combination of features to separate the classes [123]. The function should find 
the index set J = {i,,---,i,} from all possible index sets Jcanq with all feature 
candidates, at which the function values of the measure O are maximized: 


T= arg max Q(Icand) with [cand] = R. 


Icand 


The feature selection improves the model in terms of simplification for an eas- 
ier interpretation, shorter training times and avoids redundancy and overfitting. 
Overfitting is a problem occurring when the statistical model is too complex. 
It tends to describe random errors and noise instead of the underlying rela- 
tionships. If too many parameters are taken into account during the training 
phase, the classifier will learn all data by heart and will show a very good 
error on training data. On the other hand, the predictive performance and the 
generalization usually degrade and the prediction error increases. 


A multivariate analysis of variances (MANOVA) [124] can find the objective 
function Q. For the MANOVA method, each class of the output is approx- 


imately described with a n-dimensional normal distribution of the features 
x [125]. 


feature 2 
. 


feature 1 
Figure 2.6: Illustration of MANOVA. 
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The goal ofthe method is to find a group / out of n features, in which the means 
of feature values of the same class are concentrated and in which the means of 
feature values of different classes are diffuse. In other words, the means of the 
feature values of different classes are supposed to be apart as much as possible, 
which is illustrated in Figure 2.6. 


To find a measure of the described requirement, only the features from the 
feature set / are selected from the input matrix X and the new input matrix X 
has the dimension (h,n). 


The MANOVA method evaluates the feature sets based on the between classes 
variance matrix Vg and the within classes variance matrix Vw with / classes, 
which are defined as 


1 
Ve = X hj: (wur) (wir), 
j=l 
1 
Vw = Shi Ê, 


J=1 


with 


1 5 
Hy = — -Jiny Xp 


á I fa 1 ~\F f 1 = 
= (mh) j ee ? 


where u” is the mean of all features of all classes, By is the mean of features 


only describing the jth class, and Ê j is the estimated covariance matrix of the 
jth class. The total variance matrix Vr is defined as 


Vr = Vw + Vg. 
The analysis of variance implies to maximize the between groups variance 


Vg and to minimize the group variance Vw. The solution is an eigenvalue 
problem, which can be converted into an ordinary eigenproblem 
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(VB -A:Vw)v; =0 (Vw! Vg -AE)v; =0, (2.7) 


with the identity matrix E and eigenvectors v;. 


With the sorted eigenvalues (41 > 42 > --- > Aq) from (2.7), different statistics 
can be applied to find the objective function Q , such as the Hotelling-Lawley 
Trace, Pillai-Bartlett Trace, Roy’s Largest Roots, Wilk’s Lambda [126]. How- 
ever, the quality and the question, which of the statistics is the best one, is still 
in discussion, e.g. [127, 128]. Here, the Wilk’s Lambda or likelihood ratio 
test is applied, which is widely used in recent literature, e.g. [129, 130] and 
defined as 


h 1 


Ar = ——. 
2 ie 


I 


For a better interpretation of the test statistics, the test statistics is defined as 


h 
1 det(Vy) 
Ai] Je 
Qır í | 1+A; det(Vg + Vw) 


where the best value is Q = 1, which indicates a good evaluation of the feature 
set J and the worst value is Q = 0. 


The number of features to be used is set by the user, and every possible feature 
combination is tested. The best combination is selected by choosing the quality 
criterion Q closest to one. 


2.5.5 Feature aggregation 


The feature aggregation is applied to reduce the size of the feature space in 
order to compute the classification rules in a lower-dimensional space and 
to save computation time. In opposition to feature selection, no features are 
disregarded. There are several methods to perform feature aggregation as 
described in e.g. [131, 132], one of them is the linear discriminant analysis 
(LDA) [133]. 
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2.5 Data Mining 


The basic principle behind the linear aggregation is to use a projection matrix 
A € R’*" to reduce the size of the feature space from the original feature matrix 
X into the aggregated feature matrix X. 


X=XA 


The goal of LDA is to maximize the ratio between the within classes vari- 
ance Vw and the between classes variance Vz after the projection using the 
transformation matrix A. 


The maximization problem can be written as the eigenproblem 
(VB-AVw)w;=0 = (Vw 'VB-AE)w;=0. (2.8) 
The solution of (2.8) is 


A=(aj,::-,a%) with a=v;, 


where the eigenvectors v; belong to the A largest eigenvalues (A; 242--- > 
Ayn2 "2 Ay) from Vw! Vp. 


2.5.6 Performance measures 


The output of a classifier from the training and testing process is a confusion 
matrix M =(mj;) € N’ for classes K;,i = 1,...,1, as shown in Table 2.4. 


Table 2.4: Confusion matrix of a classifier from the training or testing process. 


Output class 
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In the confusion matrix, m;; presents the true positives for class i. The other 
elements in column j are called false negatives, in row i false positives and in 
the diagonal true negatives. 


An overview of various performance measures for different classification prob- 
lems to evaluate a classifier and to give proof for generalization is given in [134]. 
The most important performance measures are precision, recall and accuracy, 
which are derived from the confusion matrix from training or testing and 
calculated as follows: 


Precision; = =; 
jal Mji 
Mii 
Recall; = =; f 
I Mij 
n 
=] eii 
Accuracy = ———— 


i=1 &j=1 


2.6 Performance indices for measurement 
validation 


The goodness of fit between estimated values Y and reference values Y can be 
quantified using the coefficient of determination or R? and the normalized root 
mean square error (NRMSE). 


The coefficient of determination R? is defined as the square of the Pearson 
correlation coefficient p [135]. 


Be cov(Y,Y) 
ee: 
R= p. 


R? ranges from 0.0 to 1.0, with higher values indicating better goodness of fit. 
However, this performance index only evaluates linear relationships between 
the estimated and reference observations. Therefore, it is insensitive to pro- 
portional differences of the two data sets and an additional performance index 
to determine the true error is needed. 
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2.6 Performance indices for measurement validation 


The root mean square error (RMSE) is acommonly used measure to determine 
the goodness of fit [136] for n observations. The measure can be normalized 
with the range of the reference data, which is referred to as NRMSE. The 
values of these measures vary from 0.0 as perfect goodness of fit to co as bad 


goodness of fit. 


1 z 
RMSE = g- 


NRMSE = a Aral 
yn Y max —Ymin ` 


Since Y max and Y min are singe values, the measure is not robust for signals 
with outliers. A more robust version is the division with the expected value of 


Y: 
ılr-Fl 
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(c) NRMSE = 0.0706, R? =0.9615 (d) NRMSE = 1.8150, R? =0.2755 


Figure 2.7: Performance indices for measurement validation. 
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Figure 2.7 shows the function sin(x) as reference data and variations of 
the function as estimated observations. If the reference data and estima- 
tion are identical, R? and NRMSE show the best possible values (prettyref- 
fig:performanceindicesa). As mentioned above, R? indicates a perfect good- 
ness of fit for a proportional variation of the two data series (Figure 2.7b). 
The two time series in Figure 2.7c are very similar with a harmonic of the 
reference observations. NRMSE indicate an almost perfect goodness of fit. 
However, the variance of the two time series are identified by R?. The values 
for Figure 2.7d are poor for both indices. 


42 


3 Road condition monitoring with 
vehicle based sensors 


Outline: This chapter presents a novel concept to monitor the road infrastruc- 
ture automatically, comprehensively, inexpensively, and almost in real-time. 
Section 3.1 gives an overview ofthe concept, which is mainly based on vehicle 
sensors. The following Section 3.2 analyzes the behavior of the vehicle driv- 
ing over road features and defects. Based on these theoretical considerations, 
Section 3.3 evaluates available on-board vehicle sensors and introduces the 
selected sensors for this work. Besides the selected on-board sensors, this 
work also considers an acoustic sensor in the tire cavity to estimate the road 
condition. Section 3.4 describes a novel road condition monitoring system, 
which consists of an automated data acquisition with the selected sensors and 
data transmission to a central database. Section 3.5 introduces a novel ex- 
tended simulation approach to analyze the vehicle behavior and to investigate 
new data mining methods under controlled conditions. Section 3.6 concludes 
this chapter. 


3.1 Concept of a novel road condition monitoring 
procedure 


Figure 3.1 gives an overview of the proposed novel concept to monitor the road 
infrastructure. The vehicle sensors measure the road infrastructure directly, 
e.g. with cameras, or indirectly through the vehicle vibration due to unevenness 
of the road infrastructure. Furthermore, they measure the environment, such 
as the weather condition, which influences the sensors itself, the vibration 
behavior of the vehicle and the road infrastructure. Methods are developed 
and implemented to combine the collected sensor data from various vehicles 
and to gain information of the road infrastructure. The information, such as the 
position of road hazards or dangerous road segments, is provided for advanced 
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Figure 3.1: Overview of the data flow and participants of a novel concept for road condition 
monitoring with on-board vehicle sensors. 


driver assistance system (ADAS), e.g. to warn the driver of road hazards 
or suggest an alternative route with a smoother road surface. They are also 
visualized to allow road engineers to structure and prioritize their maintenance 
and quickly react to dangerous defects. 


The novel concept allows to monitor the road infrastructure comprehensively 
in short intervals and the road transportation departments to give an overview 
of the recent road condition. It is mainly based on standard on-board vehicle 
sensors to avoid the investment of additional sensors, which often need a special 
permit, especially if they are mounted on the outside of the vehicle. The data 
are available from the controller area network (CAN) bus and they can be pre- 
processed with electronic control units (ECUs). The required pre-processed 
data for road condition monitoring or already extracted information can be 
sent to other vehicles or to a central data bank via vehicular communication 
systems, which is referred to vehicle-to-everything (V2X) communication. 


The fundamental parts of the implementation of this concept are the selection 
of the appropriate vehicle sensors, the development and application of data 
mining methods to extract the favored information and to combine the output of 
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various vehicles as well as the output of each vehicle under different operating 
conditions, e.g. speed, wheel load, drive, or braking torque, which overruns 
the same or an adjacent location. The following sections in this chapter prepare 
for the presentation of the novel methods in Chapter 4 and the implementation 
of fundamental parts of the concept in Chapter 5. 


3.2 Vehicle motion from road irregularities 


Figure 3.2 shows the principal movement of a vehicle due to obstacles on the 
road. The red colored coordinate system with its origin as the center of gravity 
describes the orientation of the vehicle at rest, which changes its position 
during passing the obstacle, indicated by the black colored coordinate system. 
An extended obstacle on only one vehicle track mainly causes a roll motion 
of the vehicle Figure 3.2a and the change of position of the center of gravity 
in zand y direction. Examples for such obstacles are ruts on only one side of 
the road or potholes. A wide obstacles over two tracks of the vehicle are for 
example concrete joints, transverse cracks, or level crossings, which mainly 
induce a pitch motion of the vehicle Figure 3.2b. Furthermore, the original 
center of mass moves in x and z direction. The two effects can overlap, for 
example when the right front wheel of a vehicle firstly passes a pothole or the 
vehicle drives over beveled level crossings or cracks. 


(a) single-lane obstacle (b) double-lane obstacle 


Figure 3.2: Principal movement of vehicles due to road obstacles. 


The vibrations of the vehicle body are damped due to the suspension system 
(Section 2.2). The suspension compresses as the wheel moves up and extends 
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as the wheel moves down to limit the movement of the vehicle body. Since the 
tire and the suspension work as a low-pass filter, vibrations due to very small 
obstacles or the macro texture of the road might not pass through to the vehicle 
body. However, the road profile deforms the tire structure, which creates sound 
waves in the tire cavity, which is illustrated in Figure 3.3. 


velocity v wheel rim 


BEN tire cavity 
vente tire structure 


TEPE RER road surface 


Figure 3.3: Vibration of the tire due to road irregularities. 


3.3 Evaluation of on-board sensors 


Vehicles today have various sensors on-board, which also measure the envi- 
ronment of the vehicle and the vehicle vibration and movement (Section 2.3). 
For example, a stereo camera system is mounted to the inside of the windshield 
to classify objects and determine distance mainly for ADASs and autonomous 
driving [137]. The video frames can also be used to derive objects on the 
road infrastructure (Section 1.4). However, the stereo camera is currently not 
available with standard automotive packages and only with a predictive active 
suspension system, such as the Magic Body Control from Mercedes-Benz. A 
mono camera is available with less sophisticated packages but the level of detail 
to detect road defects is lower compared to the stereo camera (Section 1.4). 


The motion of the vehicle due to road irregularities (Figure 3.2) can be de- 
termined with an inertial sensor. Accelerometers in longitudinal and lateral 
direction as well as a yaw rate sensor is available with standard packages, such 
as electronic stability control (ESC) and anti-lock braking system (ABS). The 
sensors, which are important to estimate the road quality, pitch and roll rate 
sensor and vertical accelerometers are available with vehicle dynamics con- 
trol or active suspension packages. Hereby, the vertical accelerometer can be 
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placed in the vehicle body or at the front wheels. Newer vehicles, such as the 
new Audi A7 is even equipped with an inertial sensor with higher sample rate 
of up to 100 Hz as standard, whereas older vehicles only have an longitudinal 
and lateral acceleration sensor with lower sample rates to measure the driving 
dynamics. 


The motion of the suspension can be mainly measured with a ride height 
position sensor in chassis systems. 


The steering angle sensor and lateral accelerometer might be helpful to recog- 
nize evasive maneuver due to potholes. 


The position of the vehicle is necessary to map road defects. Commonly, 
vehicles have a navigation system, which provides the vehicle position as well 
as the velocity. In modern vehicles, the GPS sensor signal is combined with 
inertial sensor data to estimate the precise position with an accuracy of less 
than | m through data fusion using Kalman filtering. 


Table 3.1: Relevance of on-board vehicle sensors for road infrastructure monitoring. 


Sensor Availability Cost Relevance! 
Stereo camera Predictive active susp. high ++ 
Acoustic sensor - low ++ 
Pitch and roll rate sensor” Vehicle dynamics control low ++ 
Vertical Accelerometers~” Active suspension low ++ 
Vehicle position and velocity” Navigation system med. ++ 
Mono Camera Lane assist, i.a. high + 
Yaw rate sensor" Basic (ESC) low + 
Height sensor chassis Active suspension med. + 
Steering angle sensor Basic (ESC) low + 
Longitudinal and lateral accel. Basic (ESC) low o 
Wheel speed Basic (ABS) low o 
Radar/ultrasound Side assist, i.a. high - 
er highly relevant 
relevant 
o relevant as covariate 


= not relevant 
? at vehicle body and/or suspension system 
* considered in this work 
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Latter can also be acquired through wheel speed sensors. An on-board sensor to 
estimate the macro texture and to measure the vibration of the tire (Figure 3.2b) 
is not delivered ex factory. 


Table 3.1 summarizes selected available on-board sensors, the required au- 
tomotive package, the cost of the sensor, and their relevance to detect road 
irregularities based on previous and current research (Section 1.4). 


3.4 Novel road condition monitoring system 


3.4.1 Sensors 


The survey of related research (Section 1.4) shows that the stereo camera has 
been extensively studied in previous literature and has several drawbacks. Road 
profilometers are too expensive for data acquisition with a vehicle fleet. 


An inertial sensor in the vehicle body represents a promising sensor to estimate 
the road quality, as it measures the movements of the vehicle due to obstacles. 
An inertial sensor placed at the wheel carrier results in higher measurable 
frequencies due to lower inertia. However, since series vehicles are hardly 
equipped with inertial sensors at the wheel carrier, the focus lies on inertial 
sensors positioned in the vehicle body. Section 1.4 showed that research on 
inertial sensor for road infrastructure monitoring still has room for improve- 
ments, especially in data processing and combination of the output of various 
vehicles. 


Acoustic sensors outside of the vehicle have various disadvantages. They are 
prone to environmental influences, such as water, and suffer from acoustical 
reflections from the environment, e.g. from noise protection walls or houses, 
or noise, e.g. from passing vehicles. Acoustic sensors inside the vehicle body 
suffer from noise inside the cabin, e.g. from radio and conversations of the 
passengers. 


An acoustic sensor in the tire cavity for broad application has been rarely 
studied (Section 1.4). So far, only prototypes have been developed, which are 
too expensive for fleet tests and which need a specific vehicle permit due to 
their large constructed size and the modification of the rim for lead-through 
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of the sensors. Furthermore, the tire cavity acoustic sensor has only been 
researched in the context of tire road noise. Therefore, this work considers a 
tire cavity acoustic sensor with the aim to develop a new measuring device for 
fleet tests and to estimate the road roughness or macro texture. 


The position of the vehicle and therefore global position system (GPS) is 
necessary to visualize and map the estimated road quality or defects. The 
accuracy of the positioning can be improved by sensor fusion of GPS and 
inertial sensors, as shown in, which is implemented in modern series vehicles. 


Overall, based on the evaluation of previous research (Section 1.4) and on- 
board sensors (Section 3.3), this work considers the following sensors for road 
condition monitoring: 


e inertial sensor in the vehicle body 
e tire cavity acoustic sensor 


The following sections investigate 


e the principal graph of the raw data of the sensors, 

e if the sensors are suitable to measure the vehicle or tire vibration due to 
road damages practically and not only theoretically, 

e the signal to noise ratio (SNR), 

e the principal influences on the sensor data. 


Inertial sensor in the vehicle body 


A cleat test has been performed to investigate whether information from inertial 
sensors in the vehicle body can be extracted to detect rough road obstacles. For 
this purpose, a test vehicle BMW 116d (E87) was equipped with an inertial 
sensor in the trunk above the center of the rear axle. In addition, two inertial 
sensors were placed on the lower wishbone of the left and right wheel to 
investigate the acceleration at the unsprung masses for further research. 


Four cleats with heights of 5, 10.5, 13.0 and 19.5 mm were rolled over, with 
both vehicle lanes and with either one. The test drives were repeated three 
times with velocities of 20 and 40 km/h. The performance indices with R? of 
0.9210 and NRMSE of 0.04275 on average indicate a good reproducibility of 
the test drives [200]. 
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Figure 3.4 and Figure 3.5 show the vertical accelerations of the sprung and 
unsprung masses as well as the roll and pitch velocity of the vehicle body for 
the overrun of two cleats with different heights. 
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Figure 3.4: Vehicle and wheel response to cleat on both vehicle lanes with height 0.5 cm and 
with velocity 20 km/h. The relevant section is grayed. 


The accelerometer measures the gravity of approximately 9.81 m/s? at rest. 
The overrun for the rear axle of the cleat takes place at 2s. The grayed area 
for pitch and roll velocity of the body starts at 1.3 s, since the front axle runs 
over the cleat at 1.5 s. Figure 3.4 indicates, that the cleat with a height of 5 mm 
has a low impact on the acceleration of the body. The standard deviation of 
the signal during the overrun of the cleat is 0.062 m/s? compared to a standard 
deviation without the cleat of 0.016 m/s?. The SNR defined as the ratio of 
the variance of the acceleration at the cleat and without the cleat is 5.69. No 
significant change of the amplitude of the roll rate can be detected. The SNR 
of the pitch rate is 6.76, of the unsprung masses 6.36 on average. 


Figure 3.5 shows the sensor signals for the overrun of a higher cleat of 13 mm 
with only the left vehicle lane. The SNR of the acceleration of the sprung 
mass is higher with 9.95 as well as the acceleration of the rear left unsprung 
mass with 30.76. Furthermore, the roll motion of the vehicle is reflected by a 
SNR of 5.76 for the roll rate. The pitch rate shows a SNR of 4.84. Since the 
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Figure 3.5: Vehicle and wheel response to cleat on left vehicle lane with height 1.3 cm and with 
velocity 20 km/h. The relevant section is grayed. 


first vehicle axle passes the cleat at 1.5 s, the pitch and roll rate show a higher 
amplitude at this point. 


The analysis of all data during this test under controlled conditions with simple 
road obstacles and two different velocities allow the following conclusion: 


e The SNR for all sensor signals increase with the height of the cleat 
(compare Figure 3.4 and Figure 3.5). 

e The amplitudes of the roll rate give an indication, if the obstacle effects 
both vehicle lanes, such as a speed bump, or only one, such as a pothole. 

e The vehicle body vibration due to obstacles with a height of less than 
20 mm, which are categorized as critical by road authoraties (see Sec- 
tion 1.3), can be detected with an inertial sensor. 


Acoustic sensor in the tire cavity 


The time series of the tire cavity sound under vehicle operation with constant 
velocity of 60 km/h is shown in Figure 3.6. The data were acquired with a MB 
S-Class W220 model as test vehicle, a 255/60R16 tire, and the acoustic mea- 
surement system described in [74], where the microphone is placed inside the 
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tire and rotates with the wheel. The tire is excited by the surface texture, which 
is shown in the high-frequency wave representing the first circumferential tire 
cavity mode. 


Figure 3.7 shows the power spectral density (PSD) of the tire cavity sound under 
vehicle operation with the same vehicle (BMW 116d (E87) with 205/55R16 
tire) on different surfaces (Figure 3.7a) and with different vehicles and different 
tire sizes, a BMW 116d (E87) as vehicle 1 and MB S-Class W220 with 
255/60R16 tire as vehicle 2 (Figure 3.7b). The power spectral density (PSD) 
was calculated with the Welch’s power spectral density estimate. The peaks 
in Figure 3.7a represent the tire cavity resonance frequency and its harmonics 
(1 to 4), which can be calculated with 


l: Csound 
2n- FTC , 


i= 


where, i is the number of the harmonic, Csouna the sound velocity and rrc the 
radius of the tire cavity on average. 


Furthermore, the PSD level of both curves underline the theory that a greater 
road surface roughness leads to greater tyre deformation and thus to greater 
excitation of cavity modes within the tyre torus. The sound pressure level 
(SPL) of the tire cavity sound for the rough surface is over 20 dB higher than 
for the smooth surface. 


sound pressure (Pa) 


| | | | | l l l l 
80 100 120 140 160 180 200 220 240 260 
time (ms) 


Figure 3.6: Time series of tire cavity sound. 


The difference of the resonance frequency of different vehicles is illustrated in 
Figure 3.7b. The peaks are not at the same frequency due to the different radii 
of the tire cavity, whereas vehicle 2 had tires with a greater radius. Moreover, 
vehicle 2 had a higher velocity and a higher SPL compared to vehicle 1. 
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Figure 3.7: Power spectral density (PSD) of tire cavity sound under different conditions and with 
different vehicles. 


From the analysis of tire cavity sound pretests the following points can be 
conducted: 


e The macrotexture of the road surface has an influence on the tire excita- 
tion and the SPL in the tire cavity. 

e The frequencies of the tire cavity modes differ in the size of the tire. 

e The influence of further parameters, such as velocity, tire temperature, 
or tire pressure needs to be investigated, 

e Features have to be individually calculated for each vehicle. 


3.4.2 Data acquisition 


To acquire data comprehensively on roads with various vehicles and to enable 
a road infrastructure monitoring in short time intervals, a measuring system 
must meet the following requirements: 


e automatic data acquisition with no manual intervention, 
e automatic data transmission to a central data base in short intervals, 
e low investment per unit. 


Figure 3.8 shows the concept of a novel road condition monitoring system 
based on the selected sensors (Section 3.4). The basis of the measuring system 
is a data logger, which stores the data from various sensors during the drive. 
The sensors are an inertial sensor and GPS receiver placed in the vehicle 
body and directly connected to the data logger. A tire pressure monitoring 
system (TPMS) controls the tire temperature and pressure and transmits the 
data wireless to the data logger during vehicle operation. A telemetry attached 
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to the rim acquires the data from the tire cavity sound sensor and transmits the 
data to the data logger as well. 
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logger inertial 
y GPS receiver cee 
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A 


server 
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data processing 
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— wired communication during driving 
@ wireless communication during driving 
@ wireless communication after return to parking area 


Figure 3.8: Concept of a novel road condition monitoring system. 


The data stored on the data logger are transferred to a central database on a 
server via an access point. Access points can be placed close to the parking 
area of the vehicle. The upload starts, after the data logger is connected to 
the access point. The data from the various sensors can then be fused on the 
server, also with additional data, such as weather data. Afterwards, the data is 
processed with methods developed to provide a comprehensive and up to date 
overview of the condition of the infrastructure. 


The data logger and sensors in the vehicle are activated and deactivated with 
the ignition of the vehicle. The TPMS, telemetry, and tire cavity sound sensor 
is activated or set in stand by, dependent on the centrifugal force of the wheel. 


3.4.3 Demonstration of novel measuring device 


Modern vehicles are equipped with many sensors (Section 2.3) and the respec- 
tive data can be sampled from the CAN bus, especially the acceleration of 
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the vehicle. However, the data is encrypted and can usually only be accessed 
by original equipment manufacturers (OEMs) or automotive suppliers. Re- 
searchers have used dedicated inertial measuring units (IMUs) [138] or smart- 
phones as multi sensor platforms [139, 140]. However, dedicated IMUs are 
expensive and sophisticated to use and therefore not applicable for automated 
data acquisition with various vehicles. It is difficult to equip smartphones with 
additional sensors or to simultaneously acquire data from different positions 
within or outside the vehicle. Furthermore, the hardware can usually not be 
extended, e.g. with an antenna to improve the wireless connection to sensors 
outside the vehicle, such as the acoustic sensor in the tire cavity. 


Therefore, a new measurement device is developed to meet the requirements 
of the novel road condition monitoring system, which is defined in Section 3.4. 
The goal is to acquire data from various sensors, to enable an automatic data 
acquisition and data transmission to a central server, and to have low cost of 
investments per unit for fleet use. 


The core of the novel data acquisition method is a data logger in the vehicle, 
which consists of a single-board computer (SBC), namely a Raspberry Pi. The 
Raspberry Pi has input/outputs (I/Os) to connect additional sensors. Moreover, 
WLAN and Bluetooth adapters can be plugged in to receive or send data, as 
well as a memory card to store a large amount of data. Periphery, e.g. buttons, 
can be installed to label the data live with the desired output. The software can 
be self developed as open-source and the processor can merge and preprocess 
the data from various sensors. Furthermore, the SBC can be extended by 
additional boards, such as GPS or a uninterruptible power supply (UPS). 


Besides the data logger, a new telemetry system at the vehicle wheel has to be 
developed to transmit the acoustic data of the tire cavity to the data logger. 


The sensors of the novel road condition system are selected and briefly de- 
scribed in the following paragraphs. Furthermore, the new developed teleme- 
try system and the database to centrally store the data from all vehicles are 
specified. 


The accuracy of the selected sensors for the novel measurement device are 
validated and the results are shown in Section 6.2. The signals of the selected 
inertial sensor and acoustic sensor are compared with signals from reference 
sensors, which are accurate but too expensive for fleet use. The accuracy of the 
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velocity from GPS is evaluated by comparing the data with velocity acquired 
with on-board diagnostics (OBD). 


Inertial sensor 


Microelectromechanical systems (MEMS) sensors have a good performance, 
and have advantages in size and power consumption compared to other tech- 
nologies. Therefore, a MEMS inertial sensor (LSM9DS1) is selected to mea- 
sure the motion of the vehicles. Furthermore, the sensors measures all six 
directions of movements. 


Acoustic sensor 


In contrast to an inertial sensor, which is either available on-board via CAN 
bus, in a smartphone or dedicated IMUs, an automated and inexpensive data 
acquisition of tire cavity sound data has not yet been developed. The most 
recently developed prototype, presented in [74], to sample tire cavity sound 
data is too expensive, the battery lasts only 10 hours, the vehicle needs a special 
permit to participate in public road traffic, and it needs manual interaction to 
start and stop the measurement. 


The sensor in the tire cavity is exposed to extreme conditions. The SPL 
in the tire cavity to be measured exceeds 150 dB and the static pressure is 
approximately 2 bar above atmospheric pressure. The temperature in the tire 
cavity varies between -10 and 70 °C and the weight of the sensor influences the 
imbalance of the wheel. The frequency range to be analyzed is 50 to 5,000 Hz. 


A conventional electret microphone with low investment costs per unit does 
normally not meet the described requirements, especially to measure the high 
SPL in the tire cavity. However, a novel modification of conventional electret 
microphones is proposed to measure higher SPL with less total harmonic dis- 
tortion (THD). For this purpose, the field effect transistor (FET) is rewired from 
its common source configuration into a source follower configuration [141]. 
With the new configuration, the microphone is supposed to have a reduced 
sensitivity but an increased maximum output voltage. 
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Different types of microphones were experimentally analyzed to identify the 
sensor with the best ability to measure the tire cavity sound. The following 
microphones were considered: 


INMP411, a MEMS microphone, 

CMA-4544pf-w std., a standard inexpensive electret condenser micro- 
phone, 

CMA-4544pf-w mod., the modified electret condenser microphone with 
the source follower configuration, 

PCB 103B02, an acoustic ICP pressure sensor as reference sensor, which 
was used in [74] for tire cavity sound measurements. 


To evaluate the acoustic sensors, the following experements were performed: 


e Measurement of THD by exciting the sensor in a closed cavity with a 

sine wave with a frequency of | kHz and a SPL of 110 dB. The source 

of the sine wave has a THD of less than -80 dB (< 0.01 %). 

e Measurement of Maximum SPL by exciting a tire with the sensors in the 

cavity with an impact hammer. The tire was mounted on a seismic mass 

to avoid environmental influences and the tire pressure was 2.5 bar. 

e Measurement under vehicle operation on different types of road surfaces 
with constant speed of 30 and 60 km/h to investigate the performance of 
the sensors under the future application. 


A novel telemetry device at the wheel, which transmits the sound data to the 
data logger in the vehicle, has been developed to start and end the measurement 
automatically [142]. A 3-axis ultra-low power MEMS accelerometer is located 
on the battery-power telemetry and tracks the centrifugal acceleration of the 
wheel. As soon as a predefined threshold of the acceleration is exceeded, the 
accelerometer sends an interrupt to a micro-controller, which then enables the 
power supply for the telemetry and sends the telemetry back to stand-by if no 
acceleration is detected. 


GPS 


As GPS sensor the MTK3339 chipset is employed and connected to the data 
logger via UART to acquire the position and velocity of the vehicles. 
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TPMS 


The static pressure and temperature in the tire torus influence the tire torus 
sound and needs to be recorded. Modern vehicles have a TPMS on-board 
and direct TPMSs have a sensor usually at the valve in each tire to measure 
the pressure and the temperature [201]. Similarly to the telemetry of the 
acoustic sensor, the TPMS sensor is activated by an accelerometer. Moreover, 
a diagnostic device or the receiver of a high-line TPMS can request the sensor 
for data. The sensor is calibrated under ambient air pressure to take advantage 
of the better range of values for the analog digital converter (ADC). The 
information of the sensor, the pressure, temperature, sensor ID and received 
signal strength indicator (RSSI), is transmitted to the receiver via the carrier 
frequency of the ISM band, which corresponds to 433 MHz in Europe. The 
signal can be sampled via the CAN bus or intercepted and decoded by an 
external data logger in vehicle, which is implemented in this work. Figure 3.9 
shows the time history of the tire temperature and pressure acquired with a 
TPMS. Naturally, the first signal of the TPMS is transmitted a few minutes 
after starting the drive. The initial conditions were approximately 25 °C for the 
temperature outside and 2.4 bar for the tire pressure. 
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Figure 3.9: Tire temperature and pressure acquired with TPMS. 
Database 


The data transmission is realized via WLAN to avoid costs for mobile data 
plans, which is usually implemented when using a smart phone as data log- 
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ger [139, 140]. The data logger connects to an access point, which is for 
example close to the parking area of the vehicle, and uploads the data to a 
server, where it is stored, merged and processed. Furthermore, the software 
of the data logger can be automated updated, by pulling the software from a 
web-based hosting service and version control system, where the source code 
is managed. 
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Figure 3.10: Functional diagram of the developed data acquisition method. 


Figure 3.10 summarizes the developed method to acquire and centrally store 
the data from the sensors in the vehicle body and tire cavity. 


3.5 Extended simulation approach for road 
condition estimation 


3.5.1 Overview 


Simulation has several advantages compared to real measuring drives, e.g. a 
perfect reproducibility, low cost and effort, among others [143]. Furthermore, 
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it is difficult to investigate a single influencing factor during real test drives, 
while keeping all other dependent parameters constant. In a vehicle simulation, 
the physical behavior of the vehicle dynamics can be precisely investigated. 
Moreover, it is costly and time consuming to acquire and label real measure- 
ment data to apply supervised learning. Therefore, this section describes the 
development of an extended simulation approach for road condition estimation. 
Especially for the purpose of using the angular rates and acceleration of the 


vehicle body to estimate the road condition, the following examinations are 
considered: 


e the optimal placement of the inertial sensor in the vehicle body, 

e the optimal parameters for the data processing methods, 

e the influence of vehicle velocity and vehicle load, 

e the influence of the vehicle, suspension type, the SNR, and of the sensor 
signals on the ability to detect road features. 


FR3 TR4 


Figure 3.11: Full car model for vertical dynamics simulation. 


The consideration of all influences, parameters and constructional elements in 
the simulation is not possible due to computational costs or because the values 
for the parameters are unknown and can not be determined. Therefore, the 
vehicle model for this study sets various simplifications, which are discussed in 
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the following sections. A common full car vehicle model, shown in Figure 3.11, 
is extended to sample the vehicle body motion from different positions and 
different set-ups, such as an active suspension or anti-roll bars. Furthermore, 
the road profile as input of the simulation is extended compared to previous 
literature. 


Previous studies have focused on the behavior of the vehicle on roads with 
different levels of roughness with road models presented in Section 2.4, e.g. 
[144, 145]. However, these simulation approaches do not represent a real road 
infrastructure. This study extends previous simulation approaches not only by 
the vehicle model but also by the road model, e.g. by the implementation of 
road features and obstacles. 


The models are transferred into a state-space model as introduced in Section 2.4 
to be simulated in Matlab. The simulation can run on any modern computer 
without special hardware. The user can set the different parameters, which 
are discussed in the following sections. The simulation with a road length of 
10km takes around 5 min with standard processor (Intel Core i5) and main 
memory (8 GB). The corresponding data file then has a size of approximately 
20 MB. 


3.5.2 Extended road model 


The developed novel simulation considers not only different degrees of surface 
roughness, such as the international roughness index (IRI) as introduced in 
subsection 2.4.3. Also the following road features are implemented, such as 
defects and constructional obstacles: 


smooth asphalt road 
ralroad crossing 
manhole cover 
cobbled road 
porthole 

uneven asphal road 


The basic road surface and different levels of roughness can be modeled with 
(2.5). The implemented road features are described by continuously differ- 
entiable functions with varies parameters for height, depth, and length to 
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avoid numerical issues in the time step integration process of the simulation 
itself [146]. Thus, the transition part of the road features are smooth. The 
implemented road model can be found in [202, 147]. 


3.5.3 Extended full car model 


The basic full car model is extended by an anti-roll bar and an active suspension. 
The anti-roll bar is a part of many automobile suspensions and helps to reduce 
the body roll of a vehicle when the deflection of body and tire between left and 
right side are different [144]. The implemented model of anti-roll bar can be 
found in [144]. 


In addition to the passive suspension with basic springs and dampers, active 
suspensions use actuators between the chassis and wheel assembly, which can 
exert an independent force on the suspension in order to improve the driving 
comfort. One quarter of the full car model with active suspension is shown in 
Figure 3.12. The force f; is applied between the body and wheel assembly. It is 
controlled by feedback and represents the active component of the suspension 
system. 


Ms sensor 
Y 
I, Oi fix controller 
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Figure 3.12: One quarter of the full car model with active suspension exemplarily for wheel i. 
The force fj is controlled by feedback and represents the active component of the 
vehicle model. 
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With this model the mathematical modeling for bouncing (3.1), pitching (3.2), 
rolling (3.3) of the vehicle body and vertical motion of each wheel (3.4) is 
derived as follows: 


4 


Ms Zs = X -kss = Zui)—Csi(Zsi = ui) + fis (3.1) 
i=1 
JyyOv = > -df (ksi(Zsi = Zui) + Csi(Zsi — Zui) — fi) 
a (3.2) 
+ > dy (Ks i(Zsi — Zui) + Csi(Zsi -Zu)- Fi) 
i=3,4 
Jaxx bv = > dyi(ksi(Zsi —Zui) + Csi(Zsi — Zui) — fi) 
a 3.3) 
+ 53 —delksi(Zsi — Zui) + Csi(Zsi -Zu)- fi), 
i=2,4 


Mui Zui = Ksi(Zsi — Zui) + Csi(Zsi — Zui) — Kri(Zui —rri)— fi. (3.4) 


There has been a lot of research in the design of a suitable control strategy 
for the active suspension. [148-150] compared and investigated different 
advanced controllers, such as linear quadratic regulator (LQR), Fuzzy, Skyhook 
and Hoo. Most often the Hoo based method achieves the best results with a 
good efficiency. Furthermore, it is widely used in the automotive industry 
because of its low cost and simplicity [151]. Therefore, a Hoo controller 
is implemented for this extended simulation approach. The main control 
objectives are formulated in terms of passenger comfort and road handling, 
which relates to body acceleration, pitch acceleration, roll acceleration and 
suspension deflections Zgefi = Zsi — Zui- Hence, the feedback is 


see r 
f = &s,0v,dv,Zdefl,Zdef2,Zdef32def4) - 


Other factors that influence the control design include the characteristics of the 
road disturbance, the quality of the sensor measurements for feedback and the 
characteristics of the available control force actuator. 
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Measurement position of vehicle body motion 


The value of the acceleration is dependent on the measurement position in the 
vehicle body [143]. Itincreases with a larger distance from the vehicle center 
of gravity (CGy) along the x-axis. In simulation, the accelerations and rotation 
rates of the vehicle body are mainly measured in the CGy of the vehicle body. 
To investigate, if the measurement position of the sensor has in impact on the 
accuracy to classify the road condition, the simulation is adapted. Specifically, 
different measurement positions at (xo, yo, zo) of the vehicle body motion are 
implemented 


The relation between vertical acceleration at the new measurement position a’, 
and the acceleration at CGy a, is approximately 


a, = az +a}! =a, — by : xo + by ‘Yo, 6.5) 
for small vertical distances from the CGy. 


The vehicle body is regarded as a rigid body, hence the angle and motion is 
shared among the entire rigid body 


dy’ = By, 


bv’ = dy. 


With (3.5) the output of any position in the vehicle coordinate system can be 
approximately derived and the influence of the position of the interial sensor 
on the classification result can be analyzed. 


3.5.4 Comparison of computer-simulated and real 
measurement data 


The data acquired in computer simulation with the presented vehicle model 
are compared to real measurement data to prove the usability of the simulation 
to predict road obstacles. 


Firstly, the general behavior of the vehicle model is analyzed by overrunning a 
cleat in simulation equivalent to the real world cleat test described in subsection 
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3.4.1. Hereby, the vehicle, a BMW 116d, overruns a cleat in computer- 
simulation and real world, which is 13 mm high and 70 mm wide at the speed 
of 20 km/h. The inertial sensor is placed in the glove compartment to be 
approximately vertical above the axle. In addition, inertial sensors are placed 
at the control arms of the front axle to measure the acceleration of the unsprung 
masses. These measurement points for vertical acceleration and pitch and roll 
rate are identically reproduced in simulation. Figure 3.13 shows the result of 
this test and indicates that the general behavior of the simulation is correct, 
although the data from the real measurement shows more noise. 
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Figure 3.13: Comparison of the simulated and real vehicle response while passing a cleat. 


Secondly, the vehicle model in simulation is validated by comparing the vehicle 
vibration with real world measurement while traversing different obstacles with 
various velocities. During the real test drive six different attributes at four 
different velocities are measured. All of the following signals are acquired at 
the center of gravity. For each velocity and each attribute measure, four runs 
are conducted and then the averaged value of the parameter is reported. The 
error bars show the standard deviation of the obtained values. The result of 
the measured data and the simulation are shown in Figure 3.14. It is observed 
that the graph of the standard deviation of vertical acceleration and roll rate for 
simulation follow the graph for real measurements. 
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Figure 3.14: Comparison of the standard deviation (SD) of vertical acceleration and roll rate. 
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Figure 3.15: Relative differences of real and simulation measurements. 


The relative differences of the standard deviation values from real measure- 
ments and simulation are shown in Figure 3.15. For most of the values, the 
differences are below 25 % for the SD of vertical acceleration and for the SD 
of roll rate. The values for SD of vertical acceleration are higher for real 
measurements for asphalt and cobbled road. The values for SD of roll rate 
is higher in simulation for manhole cover for higher velocities. Overall, the 
computer simulation describes the vibration of the vehicle well, even for var- 
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ious velocities and road attributes. There are specific outliers in the relative 
differences (Figure 3.15). However, the order of the absolute values for the 
different road attributes of simulation corresponds with the real measurements 
(Figure 3.15). Therefore, the simulation is suitable for the purpose of this study 
(subsection 3.5.1), e.g. to test various parameters of data processing methods 
and investigate influences of various vehicle settings and parameters on the 
classification accuracy. However, the simulation does not perfectly replicate 
the vibration of a vehicle under all conditions. Therefore, the classifiers for 
real vehicles still have to be trained with real world measurement data. 


3.5.5 Variation of parameters for vehicle model data 
processing 


There are many parameters influencing the classification results, including the 
size and physical parameters of the vehicle, different loads on the vehicle, 
and the type of suspension e.g. passive suspension, passive suspension with 
anti-roll bar, and active suspension. This subsection describes the variation 
of parameters, which are considered for the simulation. The results of the 
investigations of varied parameters are shown in Section 6.4. 


To determine whether a classifier can be generally used on another vehicle, the 
tests of the classification with different variations are simulated. Therefore, 
a BMW 116d is simulated with additional load of 200 kg and 400 kg. Fur- 
thermore, a vehicle model with parameters compared to a S-Class W220 with 
active suspension is tested as well as with parameters compared to a Sprinter. 


Besides the variation of parameters for the vehicle model, the influence of the 
position of the inertial sensor in the vehicle body is analyzed. The measurement 
position of the vehicle body motion in the simulation is the center, axle, side, 
and corner of the vehicle, defined as S1, S2, S3, and S4. The coordinates 
(xo, yo,0) of the four positions in the coordinate system of the full car model 
are presented in Table 3.2. 
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Table 3.2: Parameters for the different positions of outputs. 


(m) SI 82 3 S4 
xo 0 1345 0 1.345 
yo 0 0 0.856 0.856 


3.6 Conclusion of novel concepts for road 
condition monitoring 


This chapter introduces a novel concept to monitor the road infrastructure, 
based on vehicle crowds and on-board sensors (Section 3.1). On-board sensors, 
which measure the environment of the vehicle sensor (e.g. cameras) or the 
vehicle motion due to road irregularities are evaluated (Section 3.2). An 
inertial sensor in the vehicle body and an acoustic sensor in the tire cavity 
are selected to be considered in this work (Section 3.3) and first data of the 
sensors are discussed. A concept of a novel road condition monitoring system 
and a simulation approach are introduced to acquire data automatically and to 
investigate specific influencing factors (Section 3.4 and 3.5). 


The concepts are the basis for novel methods to acquire sensor data, which 
includes the selection and improvement of appropriate sensor models, acqui- 
sition devices, and methods to transmit and store the data in a central data 
base. As pointed out in Section 3.1, the core of the monitoring concept is the 
development and application of novel methods to process the data to estimate 
the condition of the road infrastructure. Furthermore, methods have to be 
developed to combine the output of various vehicles to improve the accuracy 
of the prediction of the ground truth. Finally, the developed methods and novel 
measuring devices have to be implemented and verified. 
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Outline: This chapter presents methods to process data, which are acquired 
with the measuring device described in Section 3.4.3. First, the raw data is 
preprocessed (Section 4.1). Section 4.2 describes methods how to process 
data, which are either annotated with the desired output variable or without 
a label. The application of the introduced supervised learning methods for 
labeled data is presented in Section 4.3. For this application, a benchmark 
data set is acquired under controlled conditions, e.g. one specific vehicle, four 
different velocities, and three different road attributes or obstacles. Finally, 
methods are developed to find road sections, which are passed through by 
various vehicles. Moreover, methods are presented to combine the prediction 
of each vehicle with confidence z to gain a more robust overall estimation of 
the road condition (Section 4.4). This procedure is only possible for the output 
of labeled data processing, since prediction confidence r cannot be calculated 
from non-labeled data processing results. The flow-chart of the data processing 
and methods are illustrated in Figure 4.1. Section 4.5 concludes this chapter. 
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Figure 4.1: Overview of data flow. 
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4.1 Sensor data preprocessing 


Since supervised classification is applied in this study, time series needs to be 
labeled with the desired output, namely the ground truth on the road. This 
is implemented by two buttons, which annotate the data while pressing. In 
contrast to previous studies [89, 93, 94, 96], which for example label the 
data ex post with video material, the data can be annotated live. However, if 
the annotator is not experienced, the label might not be aligned with the data 
corresponding to the ground truth. Therefore, the first step of the preprocessing 
is to align the label, which needs to be done manually but can be performed 
fast since the original label gives the rough position of the related time series. 
Figure 4.2 exemplarily shows the described step. 
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Figure 4.2: Aligning of the original label. 
The button was pressed later than the attribute started, which leads to a delay 
of the label. This was manually adjusted and shifted to the correct position. 


The vehicle vibration is dependent on the velocity [152]. The velocity de- 
pendency at least for the frequency shift can be minimized by transforming 
the time series into space series [153]. Therefore, the distance increments are 
calculated with the velocity and the time vector: 


As =v-At. 


Afterwards, the space domain data is resampled to a new fixed rate, set to 
100 m-!, with a linear interpolation. The result of the resampling with linear 
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Figure 4.3: Application of different methods for resampling. 


interpolation is shown in Figure 4.3a. The figure indicates, that the new 
resampled signal follows the original one. However, overfitting can arise with 
different parameters for resampling, which is shown in Figure 4.3b. Here, 
the new signal is interpolated with splines and removes peaks of the original 
signal. 


Furthermore, Halfmann and Holzmann [154] suggest to filter signals from 
on-board sensors, which are subject to noise. For example, Figure 4.4a shows 
the resampled pitch rate from the gyroscope sensor. Despite the noise of a 
MEMS gyroscope, which is visible as spikes in the signal, it is well known 
for its good accuracy in short term [155]. There are various smoothing filters 
for different purposes, and the discussion of all possible filters is beyond this 
study. However, a suitable filter for this purpose is the median filter, which 
is robust against outliers and removes noise while preserving high frequency 
content. A nth order one-dimensional median filter for the signal x is defined 
as 


-1 -1 
y(t) = med xt - =) a+) ‚forn=2-k+l,keN. 


Figure 4.4b shows the filtered signal with n = 50, which retained the original 
shape without the spikes. 


Since there are no such spikes in the data from the microelectromechanical 
systems (MEMS) accelerometer but more noise [155], a Savitzky-Golay FIR 
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Figure 4.4: Application of median filter. 


smoothing filter is applied. The filter with the frame length n =2-k+1,k 20 
and order o < n is defined as 


n-l 


y= ), Gxt+i), 

i=- 
with n convolution coefficients C;, which are dependent on the frame length 
and the order o. The derivation of the convolution coefficients is beyond this 
study. However, corresponding coefficients can be looked up in tables, e.g. 
in [156, 157]. The filter fits a polynomial of a specified degree to frames 
of noisy data and minimizes the least-squares error [158]. Therefore, the 
filter outperforms standard averaging FIR filters, which might remove high 
frequency content with the noise. Figure 4.5 exemplarily shows the unfiltered 
and filtered vertical acceleration with n = 135 and o = 3. 


N 
A 1 
E05 
y 0 
oO 
3 -0.5 
g 1 | | 1 | | 
L 0 50 100 0 50 100 
distance (m) distance (m) 
(a) without filter (b) with filter 


Figure 4.5: Application of Savitzky-Golay filter. 
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4.2 Sensor data processing 


4.2.1 Overview 


The goal of the applied methods is to process the sensor data to estimate the 
road condition while incorporating effects, which cannot be explained by the 
output variable. For example, the tire vibration does not only increase with 
the roughness of the road but also with the vehicle velocity. The advantage of 
supervised learning methods is that various classes, such as road features or 
defects, can be defined and predicted by the model. Based on the predictions 
from the training or testing phase of the supervised model, a confusion matrix 
can be derived to estimate the accuracy and generalization ofthe approach. The 
combination of several predictions at one location is then possible based on 
the precision or confidence in the model. The disadvantages are the computing 
time of the training and manually annotation of the data. 


Therefore, in addition to the supervised learning model an approach is pre- 
sented, which does not involve training and manual data labeling. This section 
discusses the influencing effects and proposes methods to process non-labeled 
data as well as labeled data. 


4.2.2 Influencing effects 


First, the principal influencing effects ofthe vehicle body vibration is discussed 
followed by the determination of important influencing effects of the sound 
pressure level of tire cavity sound (SPLtcs)). 


Figure 4.6 shows the standard deviation of the vertical acceleration and the 
roll rate of the vehicle body driving over various road features with different 
velocities. The standard deviation is calculated for a frame length of 5m. The 
experiment was conducted four times for each velocity and road feature. The 
marker in the error bar plot represents the mean of the measurement values 
and the error bars show the standard deviation of the four observations. The 
standard deviation for the figure is multiplied by 10 for a better visualization of 
the error bars. The error bars remain small except for observations for pothole 
with velocities 40 and 50 km/h. 
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The trend of the observations indicate, that the vibration of the vehicle is 
dependent on the velocity. The vibration due to a pothole remains constant 
from 50 to 60 km/h since the wheel does not fully rebound. In contrast, a 
manhole cover above surface level increases the vibration of the vehicle body 
with higher velocities. Furthermore, the vibration from the smooth surface 
slightly increases with the velocity. Further influencing effects, such as load 
or type of vehicle, are investigated with the extended simulation approach in 
Section 3.5. 
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Figure 4.6: Vehicle vibration (standard deviation (SD)) due to road features for various veloci- 
ties. 


To determine and quantify the influence of the various variables on the 
SPL(rcs), data from an experimental study [159] are analyzed. The exper- 
iments were performed under controlled conditions with the internal drum test 
bench from the Institute of Vehicle System Technology at Karlsruhe Institute 
of Technology, which is described in various dissertations and publications, 
recently in [160]. Table 4.1 shows the full factorial variation of the parameters 
of the considered variables for the experimental study. 


The mean profile depth (MPD) of the selected asphalt is 0.56 mm and of 
concrete 1.38 mm. The sound pressure level (SPL) of the tire cavity sound 
ranges from 100 to 200 dB. The tire temperature was measured with a sensor 
placed in the tire torus. 


Regression analysis is performed to estimate the relationship of the indepen- 
dent variable (Table 4.1) and the dependent variable, SPL(rcs). Firstly, simple 
regressions are conducted and afterwards a multiple regression with all con- 
sidered independent variables. 
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Table 4.1: Considered variables and variation of their parameters. 


Surface Tire Surf. temp. Tire temp. Tire pres. Load Speed 
(°C) (°C) (Pa) (N) (km/h) 
asphalt summer 13 17 2 2400 30 
concrete winter 23 27 2.5 3600 60 
33 37 3 4800 90 


Since the data is from experiments and subject to outliers, a robust regression 
instead of a standard least squares regression is applied. An overview of robust 
estimators can be found in [161]. 


Here, a bisquare weight function with default tuning constant of 4.685 is 
performed [162]. A t-test statistics is used to test that the coefficient is not 
equal to zero but has significant influence on the dependent variable. 
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120 120 
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100 100 
40 60 80 20 30 40 
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(a) Y’ = 97.18*** +.0.20*** - x (b) Y’ = 109.59*** -0.01-x 
120 120 
m 115 115 
g 
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& 105 105 4 
100 100 
0.5621 1.3821 2,400 3,600 4,800 
mean profile depth (mm) load (N) 
(c) Ë’ = 105.28*** +2.72*** - x (d) Y’ = 105.56*** +. 0.001** - x 


Figure 4.7: Exemplary results from the investigation of the influence of various variables on the 
SPL of the tire cavity sound. Y’ represents the average regression line. 
(* p < 0.05, ** p < 0.01, *** p < 0.001) 


Figure 4.7 shows the influence of various variables on the SPL of the tire cavity 
sound and Figure 4.7a indicates that speed has the major impact on the SPL. 
The regression lines indicate the influence of the variables, whereas the data 
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points in Figure 4.7b, c, and d are colored corresponding to the velocity. The 
captions represent the average regression line. The figure suggests that the 
surface in this experiment has the second strongest impact. 


Table 4.2 shows the results from a multiple robust regression. For the regression 
the values of the features are scaled into the range of O to 1 for a better 
comparison of the regression coefficients. 


Table 4.2: Variable and corresponding regression coefficient. 


Surface Tire Surf. Temp. Tire Temp. 


2.038%% -0.35* 0.06* -0.15* 
Tire Pres. Load Speed 
12974% 1.61*** 12.00*** 


(* p < 0.05, ** p< 0.01, *** p < 0.001) 


A closer investigation of the influence of speed controlled for all other described 
variables show, that it has a non-linear incluence on SPL(rcs). Figure 4.8 shows 
the observations of the experiment [159] as well as the lines of a robust linear 
and robust non-linear regression. 


The non linear regression model better fits the observations, which confirms 
the mean square error (MSE) of 2.04 for the residuals of the non linear and 
2.85 of the linear model. 


non linear model 


linear model 


@ observations 


120 


115 
110 


SPL (dB) 


105 


100 


20 30 40 50 60 70 80 90 
velocity (km/h) 


Figure 4.8: Observations and lines of a linear and non linear regression model for SPL as depen- 
dent and velocity as independent variable with all other variables constant. 
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This finding corresponds to the results of many experimental investigations of 
the dependence of the rolling noise on the rolling speed and on the temperature. 
These have even found their way into the standard describing the legal noise 
release testing of tires (ISO 13325:2003). 


The best fit of the measured values results from a linear regression of the 
logarithm of the speed. The non linear dependency needs to be kept in mind for 
the development of a method to process the tire cavity sound data (Section 4.2). 


4.2.3 Road unevenness and roughness estimation with 
non-labeled data 


This section presents the developed and applied methods to process data of 
inertial and acoustic sensor without a training process. The advantage is that 
no labeling process is needed, which is time consuming and expensive. So far, 
no approach has been presented, which does not involve a training process or 
calibration of the vehicle. However, an absolute value for the road condition, 
such as the international roughness index (IRI), can not be estimated, and 
specific attributes, such as potholes, can not be detected without a training 
process. 


The aim of the methods is to estimate a road condition score based on the inten- 
sity of the vehicle vibration and tire vibration. Moreover, influencing effects 
on inertial sensor and acoustic sensor measurements, identified in subsec- 
tion 4.2.2, should be incorporated. The score of various vehicles for identical 
road segments should be merged to reduce outliers of single vehicles. Lastly, 
the results, namely the road condition score, is to be mapped and potential 
road hazards are to be automatically labeled. Herefore, the following data 
processing, summarized in Figure 4.9, is developed. 


After preprocessing of data (Section 4.1), features are extracted for specific 
framelengths of the vehicle trajectory from the space series, which represent 
the vibration of the vehicle or tire mainly due to road unevenness or roughness. 
Based on these features, a road condition score is calculated for each frame of 
all vehicles and all trajectories are divided into road segments. The scores from 
various vehicles in each road segment are merged to one score. Afterwards, 
a double thresholding algorithm of the road segments is applied to suppress 
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road segments in good road condition and to combine road segments in bad 
road condition. In the next, the connected-component labeling algorithm 
automatically detects and labels areas of the map in bad road condition or 
with obstacles. Lastly, these areas are exported with various information, such 
as the severity and the size of the obstacle, and the overall calculated scores, 
representing the road condition, are visualized on a map. 


distance feature ex- g building of 
: ji >| score calculation > 
series data traction road segments 


visualization 
“~ connected- 


component 


labeling 
road segments 
with obstacles 


Figure 4.9: Overview of methods to process non-labeled data. 


double 
thresholding 


Inertial sensor data 


To describe and evaluate the vibration of the vehicle due to road obstacles, the 
weighted root mean square (RMS) acceleration is extracted from the inertial 
sensor data. The RMS acceleration is described in [163]. It is expressed 
in m/s? for translational vibration and rad/s? for rotational vibration. The 
running weighted RMS takes occasional shocks and transient vibration into 
account, e.g. through potholes, and is in space domain defined by 


ı p® 
= J aw(s)?ds, 
T SOT 


where aw(s) is the instantaneous frequency-weighted acceleration and T is 
the integration space for sliding average. Since the aim of this study is to 
identify defects in the road surface, the acceleration is not weighted to keep 
all information of the signal. However, for the identification of subjectively 


aw rus(So) = 
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disturbing defects, the signal can be multiplied with a weighting function. In 
discrete form, the RMS is calculated with 


k 


1 
awrus(so)= 4] z 2 aw?lso-D). 


i=0 
with k as the numbers of data points in one frame. 


To consider vibration due to overall road unevenness and single lane obstacles 
like potholes, the sliding weighted RMS acceleration is iteratively calculated 
from the vertical acceleration and the roll acceleration of the vehicle body. 


To reduce the influence of the velocity on the vibration of the vehicle, a 
statistical data binning is applied and the running weighted RMS acceleration 
is grouped into bins of velocity intervals. The velocity bins can be freely 
defined. However, a good option is to define velocity bins representing velocity 
intervals for 30 km/h zones, inner-city roads, rural roads, and freeways. 


Firstly, the maximum and minimum of both the RMS of the vertical 
and roll acceleration are determined for each velocity bin i and vehicle j 
(min, (az rms) Max j,i(azrms) Min, i(Oxrms),maxj,i(w@xrms)). After hav- 
ing acquired a sufficient amount of data, reasonable values for the maxima and 
minima are identified. However, the values of the extrema are updated with 
new acquired data. 


Secondly, the scaled RMS for vertical and roll acceleration for small segments 
of the trajectory and for each vehicle j is calculated with 


Arms Min, (azrms) 
max; (Arms) Min, i(azrms) 
Wxrus —min;,i(Wxrus) 


max, (Oxrms)- minj (öxrms) 


scaled gg 


scaled Wxirus = 


Both scaled values are in the range of 0 to 1. To include the strongest shock 
or vibration of the vehicle due to road obstacles, the maximum of the scaled 
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RMS for vertical and the RMS roll acceleration is calculated, which is referred 
to as road unevenness index (RUI): 


azęrms Min, i(azrms) 
max j,i(dzrms)- minj i(azrms) 
Wxrus — Min, i(Wxrus) 


max;,(@xrms)—minj,i(Wxrus)) 


RUI; i = max 


The final score RUI for vehicle j ranges from 0 to 1 and represents the road 
condition, whereas 1 is bad condition. 


To reduce outliers of single vehicles, the RUIs of various vehicles for the same 
road segment are merged. For this purpose, the road infrastructure area to 
be analyzed is divided into regular quadrilaterals, as exemplary illustrated in 
Figure 4.10, and the frames of the vehicle trajectories are assigned to the cor- 
responding quadrilaterals. The resolution of the quadrilaterals, referred to as 
road segments, can be individually defined. Lastly, the RUIs of various vehi- 
cles for each road segment are combined by the arithmetic mean or maximum. 
Hereby, the arithmetic mean reduces outliers and the maximum takes the worst 
calculated RUI of one vehicle into account. Furthermore, the standard devia- 
tion of the RUIs can be calculated to evaluate the variation of the score among 
vehicles or drives. 


49.038 


49.037 


latitude 


49.036 


8.445 8.446 8.447 8.448 8.449 
longitude 


Figure 4.10: Road infrastructure divided into segments. 


80 


4.2 Sensor data processing 


To highlight road segments with a high excitement on the vehicle, which indi- 
cates a road hazard, a double threshold algorithm and a connected component 
labeling algorithm are applied. 


The double threshold algorithm accounts for noise and variation of the score 
and merges road segments with similar RUIs. The algorithm suppresses road 
segments with a score below a low threshold Tı. Road segments with a score 
above the high threshold 7) are marked as relevant segments. If the score of 
road segments with a threshold above the low threshold and below the high 
threshold is close to relevant segments, it is connected to those segments, and 
otherwise suppressed. 


The procedure is illustrated in Figure 4.11 and described with the following 
assignment procedure (4.1) for RUI of road segment S(v,u), where (v,u) is the 
location expressed by the longitude and latitude. 


S(v,u) is assigned to a region Ro, which is of no interest, or a region R2, which 
is of interest and contains only road segments with a high RUI. S’(v’,w’) is any 
connected road segment to S(v,u). The assignment procedure (4.1) ends if no 
more road segments can be assigned. A is the the identifier of the attribute. 
As(v,u) ‘= Ro, if RUI su) < Th, 
Asv,u) := Ro, if Ti < RUIs(,u) < T, and RUI 5’) > Tr, (4.1) 
Asv,u) := Ro, if RUls(v,u) >To. 


(a) three different segment 
classes (b) connect similar segments 


Figure 4.11: Double thresholding method. Black segments < T}, T} < red segments < T), blue 
segments > T). 
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The connected component labeling algorithm (Algorithm 4.1), firstly presented 
by Rosenfeld and Pfaltz [164], clusters and labels the connected road segments 
with a high score obtained from double thresholding. Finally, the scores of the 
road segments can be visualized and the clusters from connected component 
labeling represent the road hazards. 


Algorithmus 4.1 : Connected component labeling. 


Input : Structure with i columns and j rows of road segments 
S(v,u) 
Output : Matrix L with labeled clusters 


tke // count variable and temporary label 
2 L=0;,; // zero matrix with i columns and j rows 
3 for u —1toido 

4 for v — 1 to j do 

5 if S(v, u) has neighbors with similar score values then 

6 if similar neighbors of L(v, u) has labels then 

7 l = list of similar neighbor labels 

8 | L(v,u) and labels of neighbors = min(/) 

9 else 

10 L(v,u):=k 

11 | k=k+1 


The identified areas with road hazards can be summarized in a list with the 
position of the center of the cluster, the radius of the cluster and additional 
statistics, such as the minimum, average, maximum and standard deviation of 
the score over the trajectories, the number of drives through the road hazard, 
and the mean and standard deviation of the velocity. 


Acoustic sensor data 


The data from the acoustic sensor in the tire cavity is processed with the same 
method as described for the inertial sensor except for the feature extraction. 
Hereby, the SPL of the tire cavity sound is calculated to estimate the roughness 
of the road. The estimation is referred to as road roughness index (RRI). 


The SPL feature is more sensitive to the velocity and therefore finer velocity 
intervals with 7.2 km/h steps are used. Furthermore, the data set to be inves- 
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tigated is reduced by data, for which the tire temperature, tire pressure and 
load is out of a specific defined range to have comparable data and to avoid 
data acquired under extreme conditions. For example, during parking of the 
vehicle, the tire cavity adopts a similar temperature of the environment, which 
can range from -10 °C in the winter to 40 °C in the summer. However, various 
drives in different environment conditions have shown that the temperature 
stays approximately constant at 35 to 40 °C after 10 min drive. Consequently, 
the beginning of a drive and longer stops when tire temperature decreases again 
are filtered out analogous to the tire pressure, which approximately follows the 
tire temperature (Figure 3.9). The remaining data is then processed according 
to the flow chart in Figure 4.9. 


Trend identification 


A trend over time of the estimated RUI or RRI can be identified by applying a 
regression over time. A robust regression is applied with the same parameters 
as described in subsection 4.2.2 for the RUI or RRI merged over all vehicles 
for each road segment. Figure 4.12 shows two scatter plots of the score of RUI 
over time for two road segments from the same vehicle. On the x-axis, day 0 
corresponds to 8th December 2016. Figure 4.12a indicates, that the score for 
the road segment increases, which suggests a degradation the road condition. 


This indication is underlined by the estimated regression function with a highly 
significant positive regressor. Figure 4.12b shows a slightly decrease of the 
score. This method presented here can be applied to a large data set. The trend 
can be visualized on a map and the road segments can be colored accordingly 
to the value of the calculates regressor. 


0.4 E 0.4 5 
= 0.35 0:39 
~ 0.3 ~ 0.3 
0.25 | | 0.25 © | 
0 100 200 0 100 200 
time (days) time (days) 
(a) Y’ = 0.3020*** + 0.00015*** - x (b) Y’ = 0.3199*** —0.00007* - x 


Figure 4.12: Examples for trend identification. 
(* p < 0.05, ** p < 0.01, *** p < 0.001) 
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4.2.4 Road attribute and surface estimation with labeled 
data 


Labeled data allow the detection of specific objects or road surfaces with 
specific characteristics via supervised classification. The road roughness is 
estimated with supervised regression. The most important part of the process- 
ing of labeled data is to find features, which can separate the desired classes 
in the feature space. Firstly, candidates of features are extracted based on 
theoretical and physical considerations or the analysis of data from pretests, 
which is referred to as feature engineering. Another option is to use feature 
learning techniques, to automatically find representations of the raw data for 
classification [165, 166]. However, the latter approach is not considered in this 
work. 


The feature candidates are automatically evaluated with multivariate analysis 
of variances (MANOVA) (subsection 2.5.4) and the best features to separate 
the classes are selected to reduce the dimension of the feature vector and to 
avoid overfitting. Overfitted models have a poor generalization with too many 
parameters compared to the number of observations. Then, the model rather 
describes random effects than the underlying relationship. 


It is difficult to illustrate the high dimensional feature space. Therefore, the 
dimensions can be further reduced by feature aggregation, for example, with 
discriminant function analysis (subsection 2.5.5). Therewith, the aggregated 
feature space with for example two dimensions, can be visualized. The dis- 
advantage of this method is a decrease of accuracy of the classification and 
the difficult interpretation of the aggregated features. Finally, the regression 
or classification is performed with a support vector machine (SVM) (subsec- 
tion 2.5.2 and 2.5.3). 


This work differentiates between the classification of road attributes, such as 
potholes, and the estimation of the road roughness within the macro texture. 
The object detection is based on the vibration of the vehicle body measured 
by the inertial sensor while the road roughness estimation is based on the 
tire vibration captured by the tire cavity sound (Section 3.4). Therefore, the 
first data set consists of the acceleration and angular rates of the vehicle in 
three directions and the second of sound pressure in the tire cavity. Both 
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(1) a,W,Vy,... 
: (2) prc. vv»... 


feature engineering 


data sets are extended by influencing parameters, such as the vehicle velocity 
(subsection 4.2.2). 


The methods to process labeled data are summarized in Figure 4.13. The raw 
data is preprocessed and transformed into space domain to extract suitable 
features. 


MANOVA 


distance 
series data 


feature ex- 
traction 


|- 


feature selection 


| 


classification/ feature ag- 
output <x A ; 
regression gregation 
discriminant 
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(1) road type, 
: attributes 
: (2) road roughness 


function analysis 


Figure 4.13: Overview of methods to process labeled data. 


Inertial sensor data 


e vertical acceleration az, 


e roll rate wy, 
e pitch rate wy. 


Afterwards, the best features are selected with MANOVA and aggregated with 
discriminant function analysis. To finally estimate the road type or attributes, a 
SVM classification is applied and a regression to estimate the road roughness. 


For the feature extraction of inertial sensor data the following data streams are 
considered: 
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Furthermore, the roll and pitch acceleration as well as the jerk are deviated in 
time domain of the above data streams. The jerk å; for step t is calculated with 


= az(t) — az(t- 1) 
—— 


åz(t) 


The time series is transformed into space domain, as discussed in subsec- 
tion 4.2.2. Furthermore, the data is transformed into spatial frequency domain, 
which contains the short-term, space-localized frequency content of the signal 
[167]. The spectrogram for a signal x with window function w(s), spatial 
frequency w and space index T is expressed in the discrete path case as 


oo 


spec{x($)}(7,0)= I, xIklwik -rle 


k=-» 


Hereby, features based on specific frequency bands can be investigated. In 
addition, the spectral centroid from the spectrograms of signal x for each time 
index Tt is calculated with 


Di m(k)w(k) 
DL, m(k) 


where m(k) is the magnitude of frequency bin number k, w(n) the representing 
frequency, and K the number of frequency bins [168]. The spectral centroid 
is a quantification of the power distribution of the signal [169]. Overall, the 
three data streams are extended by the following data streams, which leads us 
to 10 data streams in total: 


centr{x(s)HT) = 


vertical acceleration: a; (1), 

roll acceleration: w, (2), 

pitch acceleration: wy, (3), 

jerk: az (4), 

Fourier transformed vertical acceleration: spec{az} (5), 
Fourier transformed roll acceleration: spec{ wy, } (6). 
Fourier transformed pitch acceleration: spec{w, } (7), 
centroid of spec{a;} (8), 

centroid of spec{ wy } (9), 

centroid of spec{w, } (10), 
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The features are calculated for frames with a specific length in spatial domain 
and a specific overlap. If a longer frame length is chosen, short peaks for 
example due to potholes have a weaker impact on the value of features, which 
incorporate the overall signal, such as standard deviation. These short ampli- 
tudes can be captured by shortening the frame length or using features, which 
calculate extrema. 


From the data streams (1) - (4), the following features are calculated 


e average: X, 
e standard deviation: or, 
e peak-to-peak value: p, 


with 


„Glen? 
o(x) = — WI `’ 


p(x) = max (x) -— min (x), 


where x is a data frame of the corresponding data stream (1) - (4) and N the 
framelength. 


The effective value or RMS (Xgms(spec)) is calculated for the short-time Fourier 
transformed data streams (5) - (7) for the following spatial frequency bands 


(1/m) 
[0.1, 0.5] [0.5,15] [15,20] [0.1,25] [0.1, 50]. 
Afterwards, the following features are extracted for each frame of the vehi- 


cle trajectory from XrMScspec) Of each frequency band and from the spectral 
centroids (8) - (10). 


e average: XRMS(spec) and Xcentrs 
e standard deviation: ORMS(spec) and Ccentr, 
e maximum: MAXRMS(spec) ANd MAX centr. 
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Subsection 4.2.2 discusses the strong influence of velocity on the vehicle 
vibration. Previous research suggests to perform a linear regression with 
each feature as the dependent variable and the velocity as the independent 
variable [94]. 


The velocity dependency is then reduced by subtracting the estimated lin- 
ear equation from the corresponding feature. However, the vehicle vibration 
and the extracted features are not linear dependent on the velocity (subsec- 
tion 4.2.2). 


In this work, the dependent parameters are incorporated in the classification and 
therefore, mean velocity is calculated for each window as additional feature. 
To allow non-linear relationships a polynomial kernel function of higher order 
is applied on the classification (subsection 2.5.2), which is defined as 


k(x,y) = (x7 y)4, (4.2) 


where x and y are features and d the degree of the polynomial. 


Acoustic sensor data 


The described data processing method for the inertial sensor data is analogous 
applied for the tire cavity sound data. Road surfaces with various characteristics 
especially with different grain size distributions can be classified based on 
power features of different frequency bands, as we have already shown in [170, 
171]. 


This work pursues to estimate the road roughness based on the tire vibration 
represented by the SPL of the tire cavity sound and applies a SVM regression. 
Subsection 4.2.2 shows that the velocity has also a strong influence on the tire 
cavity sound pressure and is incorporated as a feature for this regression. 


Furthermore, the tire pressure, tire temperature, and vehicle load is included 
although the influence is weaker compared to the velocity. 


Overall, the RRI can be estimated for each window with the following regres- 
sion function 


RRI = ßı -SPLrcs) + 82-0 + B3- p+ Ba-T + ps- P + Bo, 
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with the following features: 


e SPLircs): sound pressure level of the tire cavity sound, 
e ý: mean of the velocity, 

e p: mean of the tire air pressure, 

e T: mean of the tire air temperature, and 


e P: mean of the vehicle load. 
The features are scaled to the range of 0 to 1 with 


, x-min(x) 


~ max(x)—min(x)’ 


where x is the feature and x’ the scaled feature. Furthermore, a polynomial 
kernel function of higher order as presented in (4.2) accounts for the non-linear 
relationship of velocity and SPL(rcs). Further features can be integrated, such 
as the type of tire, temperature of the environment, if water or snow is on the 
road, among others. 


4.3 Exemplarily application of methods on 
benchmark data set 


A benchmark data set is used to exemplarily show the data processing with the 
introduced supervised learning methods (Section 4.2) with inertial sensor data 
of the vehicle body. Moreover, the influence of classification parameters, such 
as the kernel trick or feature aggregation, and the performance measures are 
explained with this data set. 


The benchmark data set is designed for good illustration and less complexity. 
Therefore, only three features are extracted to separate three classes, which 
are potholes, surface with light damages, and smooth surface. The extracted 
features are standard deviation of the vertical acceleration, roll and pitch rate, 
which represent the principle vibration of the vehicle. 


The data was acquired on different road segments consisting of the mentioned 
road features with a BMW 116d. To further reduce the complexity of the 
benchmark data set, the road features were overrun with specific velocities, 
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| E potholes Ü E damaged surface Ü E smooth surface 


which are 30, 40, 50, and 60 km/h. The distribution of the values of these 
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Figure 4.14: Distribution of feature and velocity values of benchmark data set. 


features and the velocity for the acquired data is shown in Figure 4.14. 
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Figure 4.15: Single feature space of benchmark data set. 


4.3 Exemplarily application of methods on benchmark data set 


The histograms indicate, that the number of data points per velocity is ap- 
proximately uniformly distributed (Figure 4.14a). The values for data points 
corresponding to potholes are mainly larger over all features compared to other 
data points (Figure 4.14b, c, and d). 


The standard deviations for damaged and smooth surface overlap for some data 
points. Therefore, we can expect wrong predictions from the classification of 
these features. 


Figure 4.15 illustrates the classes in features space. Potholes are separable 
from the other classes, whereas damaged and smooth surface overlap. 


Table 4.3 shows the confusion matrix of the classifier, a standard linear SVM 
with penalty term C = 1000 and e = 1. Wrong predictions occur only between 
the classes damaged and smooth surface. 


Table 4.3: Confusion matrix from single feature classification. 


Output 
1 2 3 
z 1: Smooth surf. 820 10 0 830 
2 2: Damaged surf. 2 1021 0 1023 
m 3: Potholes 0 0 34 34 
822 1031 34 1887 


Performance measures of the training process and 5-fold cross-validation, 
derived from the confusion matrix as explained in subsection 2.5.6 are listed 
in Table 4.4. Overall the accuracy of the cross-validation is 99.0 %. 


Table 4.4: Performance measures of single feature classification. 


Training Crossvalidation 


Precision Recall Precision Recall 


(%) (%) (%) (%) 

1 99.8 98.8 99.3 98.4 

2 99.0 99.8 98.8 99.4 

3 100.0 100.0 100.0 100.0 

Average 99.6 99.5 99.3 99.3 
Accuracy 99.2 99.0 
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The high dimensional feature space is reduced to two dimensions with two 
features aggregated with linear discriminant analysis (LDA) function (subsec- 
tion 2.5.5). Figure 4.16 shows the two-dimensional feature space. The solid 
lines represent the hyperplane to separate the classes and the dashed line shows 
exemplarily the maximum margin of the support vectors and the hyperplane. 
Figure 4.16a shows the hyperplane for the linear SVM and Figure 4.16b for 
the nonlinear SVM with polynomial kernel for degree-2 polynomials. For this 
example, the nonlinear SVM creates no big advantage. However, for a larger 
data set with various classes and features, it might be necessary to increase the 
accuracy of the classifier. 


e potholes e damaged surface » smooth surface 
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Figure 4.16: Aggregated feature space of benchmark data set. 
Table 4.5: Confusion matrix from aggregated feature classification. 
Output 
1 2 3 
m 1: Smooth surf. 819 11 0 830 
2 2: Damaged surf. 14 1009 0 1023 
2 3: Potholes 0 0 34 34 
833 1020 34 1887 
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Table 4.6: Performance measures of aggregated feature classification. 


Training Crossvalidation 


Precision Recall Precision Recall 


(Io) (To) (%) (To) 
98.3 98.7 98.5 98.7 
98.9 98.6 98.9 98.7 
100 100 96.9 100.0 
Average 99.1 99.1 98.1 99.1 
Accuracy 98.7 98.7 


Table 4.5 and 4.6 indicate that the values of the performance measures de- 
crease with the aggregation of the features. Therefore, if computation time of 
classification is not critical, a feature aggregation should be avoided. 


4.4 Multiple vehicle fusion 


4.4.1 Overview 


The vibration behavior of various vehicles differ greatly due to the structure, 
suspension system characteristics of the vehicle, and the position of the measur- 
ing system. Each vehicle might have an individual classifier based on different 
feature spaces. This section addresses two problems, which are published 
in [172, 173]. 


Firstly, trajectories or trajectory frames from various vehicles through the 
identical road segments are found based on distance algorithms. Four distance 
algorithms are presented in subsection 4.4.2 and the results and computing 
time of the algorithms are compared. In contrast to other methods, such as RŽ, 
the distance algorithms find similar trajectory frames more precisely. Hereby, 
trajectories pointing in other directions are not considered as similar. This 
is essential to ensure accurate training data for new vehicles, which can be 
acquired automatically by labeling the data based on the label of the road 
segment from ground truth data of other vehicles. 
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Secondly, the classification results from different vehicles for one identical 
road segment can be merged with different methods to achieve a combination 
with the highest confidence. For this purpose, different combination strategies 
are presented and investigated in various scenarios, which are outlined in 
subsection 4.4.3. 


4.4.2 Identification of road segments with trajectories from 
different vehicles 


To find identical road segments, which are overrun by various vehicles, the 
trajectories from those vehicles are compared. Firstly, arange search algorithm 
finds the trajectories from vehicles, which are close to a reference trajectory. 
Trajectories Ar and Br of the same length are time series of global position 
system (GPS) coordinates in the form [174] 


Ar =((a1,a7),(a3,a3),....(a},a7)), 


Br =((b} ,b, ),(b3.b3),..,(bn.bn)), 


where a* and b* represents the longitude coordinate, a” and b” the latitude 
coordinate and n the number of GPS coordinates representing the trajectory. 


To find trajectories, which are close to each other, a range search algorithm 
is applied to reduce the computing time of the overall procedure. The range 
search algorithm, which is based on a k-nearest-neighbor (knn) algorithm, is 
to find all points P4 within a radius around a point Pp [175]. A k-dimensional 
(kd) tree is constructed out of all points with Algorithm 4.2, which assigns 
each point to a node to have an equal distribution of number of points in each 
node, d represents the depth of the tree. Figure 4.17 exemplary shows a 2d 
tree with 100 points and 8 nodes, each consists of 12 to 13 points on average 
and the application on trajectories, where the points are represented by GPS 
coordinates. 


After the construction of the tree, the points can be identified, that lie within 
a circle of radius r around a new point Pp. The corresponding algorithms are 
explained in more detail in our paper [172]. The basic idea is to find all nodes, 
which share an area within the circle around point Pp. In the second step, only 
these nodes are searched for points, that have an Euclidean distance smaller 
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reference points 
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(a) a kd tree with lines, which separates the (b) application of the range search algorithm 
points into 8 nodes with 12 to 13 points to find trajectories close to the reference 
on average trajectory 


Figure 4.17: Illustration and application of the range search algorithm. 


than r to the point P,,. With this method the time complexity in big O notation 
can be reduced from O(n) for common search algorithms to O(logn). 


However, as Figure 4.17b indicates, also coordinates from trajectories that 
run in a different direction as the reference trajectory are recognized as close 
points. The transferring of labels or fusion of classification outputs lead to 
wrong results. Therefore, further algorithms must process the trajectories 
to narrow the candidates of GPS coordinates belonging to the identical road 
segment and road lane. 


Methods, which can follow the range search algorithm and eliminate wrong 
points are the Euclidean distance, principal component analysis (PCA), haus- 
dorff distance, and dynamic time warping (DTW) distance. 


The Euclidean distance is defined as 


N 
1 
Di= xD la + (anon), 


n=l 
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where the length of the compared segments of trajectories Ar and Br needs to 
be equal. With the Euclidean distance, only segments of trajectories with the 
same driving direction and therefore on the same driving lane are determined, 
as shown in Figure 4.18a. 


Algorithmus 4.2 : Build kd tree. 


1 [tb] Input: A set of points P, current depth d, maximum number 
of points in node n 
Output : The root of the kd-tree storing P. 


2 if P contains only less than k points then 

3 Return a leaf v storing this point 

4 else 

5 if d is even then 

6 | dimex 

7 else 

8 L dim y 

9 l — median(Paim) // Lis the line, that splits the set of 


points at the median in dimension dim. 
10 for All points p; in Pdo 
11 if Pi,dim < l then 


12 | add p; to Pı 
13 else 
14 L add p; to Pa 


15 Vleft — buildKdTree(P), d+1) 

16 Vright — buildKdTree(P2, d+1) 

17 Create a node v storing /, make vze fr the left child of v, and 
make v,ight the right child of v. 


ıs Return v 


The PCA converts the points into a set of points of linearly uncorrelated prin- 
cipal components a, and by. of trajectory Ar and Br respectively. Afterwards, 
the Euclidean distance of the coefficients is calculated, whereas a smaller 
distance indicates a greater similarity of the two trajectories [176]: 


2 
Do = () (ag - bg)? 
k=] 


Figure 4.18b shows the green colored trajectory segments, which were identi- 
fied by the D2 function as close to the red trajectory. 
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(a) Euclidean distance algorithm 
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(b) PCA algorithm 


Figure 4.18: Application of range search algorithm with Euclidean distance and PCA algorithm. 


After the application of the PCA algorithm, the trajectory segments that turn 
on the interaction and point into the orthogonal direction of the red trajectory 
are not detected as close. However, the trajectory segments with an opposite 
direction are incorrectly detected. 


To address this problem Algorithm 4.3 is applied to consider the driving 
direction of trajectory Ar and trajectory Br and to get same results as for 
function D,. 


Algorithmus 4.3 : Consider direction. 


Input : x,y, Type: section of trajectory 
Output : sameDirection, Type: boolean 


ı latitudeFactor — (x(end)jarirude -X(start)iaritude)* 
(ylend)tatitude -y(start)jatitude) 
2 longitudeFactor — (x(end)jongitude ~ X(start)jongitude)* 


(ylend)iongitude - y(start)jongitude) 
3 if latitudeFactor > 0 AND longitudeFactor > 0 then 


return true 
else 


e 
4 | 
5 

6 L return false 
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The Hausdorff distance calculates the spatial distance between two trajectories 
as [177] 


D3 = max{d(Ar,Br),d(Br,Ar)}; 
where d(Ar,Br) = MaXaeA- Minpep- lla- b||. 


The application of the Hausdorff distance shows similar results to the PCA 
distance, where trajectory segments with an other direction could not be filtered 
(Figure 4.18b). 


The DTW algorithm finds the minimum comprehensive path between two 
trajectories, which minimizes the cost of the warping. The distance is defined 
as [178] 


bo, a 
D4 = min{ 5) Wn)? }, 


n=1 


where w, is the nth element of the warping path. The DTW algorithm returns 
similar results as the Euclidean distance (Figure 4.18b). 


Calculation time of distance algorithms 


Since all distance algorithms, partly by applying the direction Algorithm 4.3, 
produce similar results, the algorithm is selected based on runtime. Table 4.7 
shows a comparison of the algorithms with respect to the measured time 
per 1,000 calculations. The Euclidean distance is outperforming the other 
algorithms due to its simple computation. 


Table 4.7: Average run time of the algorithms for 1,000 trajectories with a length of 50 or 200 m. 


Algorithm Time (s) 
50m 200 m 
Euclidean distance 0.0050 0.0105 
PCA distance 1.0170 1.1600 
Hausdorff distance 1.1612 4.8872 


DTW distance 0.0598 0.0708 
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Calculation times with and without range search algorithm 


The range search algorithm function is intended to accelerate the algorithm, 
since all nearby points can also be found using the algorithms discussed in 
subsection 4.4.2. A test is performed with a data set of 215 km travel distance 
and 251,177 data points that are up to 70 km apart from each other. The 
method to find close trajectories is applied with and without the range search 
algorithm. The results in Table 4.8 show, that the use of the range search 
algorithm is indispensable. The calculation time with this algorithm is almost 
500 times faster than the Euclidean distance algorithm alone. 


Table 4.8: Run times with and without range search algorithm (RSA) for a traveled distance of 


215km. 
Time RSA (s) Time Euclidean (s) Sum (s) 
Without RSA - 590.1277 590.1277 


With RSA 0.4607 0.6290 1.2358 


The performance of the Euclidean distance algorithm with and without range 
search algorithm is also tested with a high density of trajectories in a small 
area. Therefore, the intersection data, which has been used in Subsection 4.4.2, 
is multiplied 50 times. Consequently, there are 1,050 trajectories with a total 
of 48,350 data points in a small area. A test with many vehicles can therefore 
be simulated. If each car passes an intersection twice a day, the data set 
corresponds to approximately 18 vessels, which navigate the crossing in 30 
days. The results in Table 4.9 show that the Euclidean distance algorithm 
performs well even at a high density of trajectories. The calculation time is 
even smaller including range search algorithm in the pre-processing. 


Table 4.9: Calculation times in a dense area. 


Time RSA (s) Time Euclidean(s) Sum (s) 


Without RSA - 1.8578 1.8578 
With RSA 0.0675 0.3853 0.4528 
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Accuracy of the transfer of ground truth data 


The goal is to communicate locations together with the corresponding road 
attribute, where an other vehicle has recognized road attributes. Thus, the 
second vehicle has training data to learn a correct classification of the char- 
acteristics based on its own measurements. To test the method to transfer 
ground truth data to train a new vehicle, an intersection with trajectories from 
the BMW 116d test vehicle is used. A single road infrastructure attribute, 
namely a railway crossing, is approximately at position (8.4465, 49.0365). 
The corresponding data from the BMW test vehicle are annotated with railway 
crossing. To show the functionality and accuracy of the method to transfer 
ground truth data, railway crossing is overrun multiple times with a new test 
vehicle, a Smart. 


Figure 4.19 shows the result of this test. The green sections are the actual 
railway crossings from the ride with the Smart and the blue sections represent 
data with the automatically transferred label railway crossings from already 
existing BMW 116d data. The figure shows that the label railway crossing 
is successfully transferred except for one case. The unsuccessful transfer is 
due to bad GPS signal and consequently the route is too far away from the 
previously traveled routes. However, this underlines the motivation to only 
transfer ground truth data, if the conditions, e.g. the GPS signal, are well and 
if there actually was an anomaly in vibration of the vehicle. 


mums actual attributes transferred attributes old points 


49.0365 


latitude 


longitude 


Figure 4.19: The result of the test to transfer ground truth data to new vehicles to train a new 
classifier. 
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With the described method in subsection 4.4.2, the label of already existing 
training data can be transferred to data from a new vehicle with a different 
suspension system to develop a new individual classifier. Labels are only 
transferred if a certain amount of labels from different vehicles and classifiers 
exist. Furthermore, labels ofroad hazards, e.g. potholes, are only transferred if 
the signal ofthe new vehicle actually shows an anomaly. Therefore, the amount 
of labels, which are wrongly transferred, decreases. Without the mentioned 
conditions, the transferred label might not be correct in the following situations: 


e Drivers of an untrained vehicle might avoid to drive over a specific 
hazard, e.g. a pothole. However, the road segment with this hazard was 
already labeled as pothole, since earlier drivers overran this attribute. 
Therefore, the data of the untrained vehicle corresponding to this road 
segment might automatically be labeled as pothole although the signal 
shows no anomaly. 

e Road hazards, such as potholes, might be repaired meanwhile and are 
not present anymore. 

e The previous collected label for training data is wrong or the classifica- 
tion algorithm predicts a wrong attribute. 


4.4.3 Multiple vehicle combination 


Since functions Dı to D4 find similar trajectories on the same road lane, the 
output for a specific road segment form various vehicles can be combined 
with the following method. The precision presents the fraction of retrieved 
instances that are relevant and can be seen as the probability 7; of the classifier 
to predict class i as class j for i,j = 1,...,1. 


A precision matrix Il can be derived from the confusion matrix for each vehicle 
m with k overruns as shown in Table 4.10. 


The precision vector for class K; for vehicle m is then defined as 


„® = 


(m) 
J yes 


(Tij -oij 


The approach, illustrated in Figure 4.20, is to combine the prediction from 
various vehicles or overruns with prediction probability 7; for a specific road 
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Table 4.10: Precision matrix derived from the confusion matrix. 


Output class 
Kı Ka ae Kı-ı Kı Total 

R Kı a 712 “in R(I-1) Til 1 
8 Ky T21 T22 ses TX1-1) Ta 1 
o 
2 f 
= : 
=E Kia | aa- T(1-1)2 ga TA(-1)X(1-1) Ta- 1 

Kı a 112 sae T(1-1) a 1 


segment with the median position (long., lat.). The goal is to get a robust 
and more precise estimation of the corresponding ground truth, in contrast to 
consider the output of only one vehicle or a simple maximum vote. 


classification by each vehicle fusion of all 
classification results 


vehi.) —————> class KY 


veho) ————> class KP a 


f 


„® 
J 


EN 


road segment Jong. lat.) : : final prediction 


N | | Re 
a j 


= (n) 
veh(n—-1) ———> class KẸ) Tj 


veh(n) ————> class K” 


Figure 4.20: Approach to combine the predictions of the condition of road segments from vari- 
ous vehicles. The final prediction has the highest confidence by incorporating the 
confidence 7 of the predictions of each vehicle. 
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Vehicles might have different classifiers with different precision vectors. Also, 
the actual precision vector of a vehicle might be unknown but can be approxi- 
mately estimated through the testing process of the classifier. 


Combination strategies 


Since the prediction of a vehicle usually has an uncertainty, multiple vehicles 
can be combined to increase the confidence of the prediction compared to only 
one vehicle, which is regarded as multiple expert problem. The final decision 
for each road segment can then be made by the combination of the individual 
predictions [179]. The classifier predicts one out of / possible classes and from 
the precision matrix zr, the confidence of the corresponding classification be 
be derived. 


The collected predictions for one road segment, named observation o € R’, are 
distributed according to a multinomial distribution. The observation o € R! 
counts the number of predictions per class. 


The following methods provide a final estimation of the condition of one road 
segment. One possibility is to count how often each attribute is predicted, 
which is referred to as majority vote. The one with most votes is assumed to 
be the ground truth 


Vinajority = arg max Oi. 
1él,..., 


An extension to the majority vote approach is the vote with the diagonal vector 
from the precision matrix, which is referred to as vote with precision diagonal 
matrix. Here, the counts for the predictions are weighted with the precision 
vector: 


Wr = (N11,722,.. 711). 


That leads to an adapted observation o’: 


o' =owr.. 
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As before, the final decision is class with the most votes: 


/ 
Vaiagonal = arg max o`. 
iél,...,1 


Each of the n vehicles might use an individually optimized classifier. This 
leads to a precision matrix II) and observation o for vehicle k. Due to 
the assumption of independent predictions, the probability that class i is the 
ground truth, is calculated as 


bo) 
Mamae 


1 n Tq peo’ 
q=1 lk=1 jag) T 


Poli = 


with O = {o} for k = 1,...,n [179]. 


As final estimate, the class with highest probability is selected with 


Vnatrix = arg ‚Dax Pale 
L 


Peer} 


The latter vote is referred to as vote with precision matrix. 


Scenarios 


The presented methods to estimate the condition of a road segment are eval- 
uated by simulating different scenarios. The proposed scenarios considers 
various situations, e.g. that the road condition changes after maintenance or 
that a classifier of one vehicle is over-fitted and outputs random predictions. 
The point of interest is how many vehicles or overruns are necessary to reach 
a very small proportion of misclassifications for the prediction of the road 
segment and how to react to latter described scenarios. 


Scenario 1: Classification with one global precision matrix In Scenario 1, 
one global precision matrix II, for all vehicles is considered. For simulation, 
a road with a specific number of segments, each segment having a different 
attribute, is assumed. The outputs of various vehicles lead to the evaluation 
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of the different fusion strategies. Additionally, the prediction quality of the 
different classes is analyzed. 


Scenario 2: Classification with unknown vehicle precision matrices In 
reality, not all vehicles use the same distribution for prediction and the real 
precision matrix is unknown, since it might be different to the precision matrix 
calculated during training or testing. Therefore, in the second scenario each 
vehicle classifies the road segments by its own classifier while the final decision 
is made based on the global precision matrix mp. Hereby the vehicle precision 
matrix nk) is an slightly adapted version of the global one. In the simulation, 
vehicle k generates a random number to classify the road segments. For 
segment having property i, it follows the multinomial distribution described 
by the entries 7;;(k),j = 1,...,/ from the adapted precision matrix r”. 
Scenario 3: Classification with equally distributed precision matrix An- 
other scenario under consideration is the classification by a vehicle totally 
independent of the ground truth. An example in real life is a vehicle with a 
classifier, which is over-fitted, a change of the vehicle vibration behavior or a 
broken sensor. For this scenario, the effect on the fusion strategies, the possi- 
bilities to identify the vehicle making trouble, and to react with adjustments of 
the combination method are analyzed. 


Scenario 4: Change of the road condition As last scenario the change of the 
ground truth from one time to another is simulated. It characterizes situations 
like a pothole that gets repaired, or a hard winter that destructs the road. It is 
investigated if rules of forgetting or less weighting the past can improve the 
outcomes. 


The calculations and investigation of multiple vehicle fusion with 10 different 
vehicles are implemented in Matlab. In the calculation, the road consists of 
s segments and each segment represents one of / classes as ground truth with 
l = 5 for all scenarios. In reality, theses classes could correspond to types of 
damages, such as potholes. Therefore, the number of wrong predictions ranges 
from 0 to s for the created road. The values of the global precision matrix 
are set as shown in Table 4.11. This precision matrix is calculated from a 
confusion matrix based on a testing process from a real vehicle. 


Each vehicle drives over the road and classifies the segments with its individual 
precision matrix, which is unknown to the back-end. 
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Table 4.11: Global precision matrix for simulation. 


Output class 
Ki Ko K3 K4 Ks Total 
Kı 0.9 0.02 0.03 0.01 0.04 1 
E Kə 0.1 0.8 0.05 0.02 0.03 1 
2 K3 0.07 0.03 0.82 0.04 0.04 1 
5 K4 0.001 0.067 0.023 0.822 0.087 1 
Ks 0.13 0.02 0.03 0.09 0.73 1 


For Scenario 1 to 3, 5 segments are simulated and 10,000 iterations of the 
drives are performed to reduce stochastic effects. In Scenario 1 the precision 
matrix of the vehicles correspond to the global precision matrix. The total 
number of wrong predictions are counted and the proportion of wrong classi- 
fications of all classes are analyzed as well as the wrong predictions of each 
class and number of vehicles. The wrong predictions in scenario 1 are calcu- 
lated with the combination method based on simple vote, vote with precision 
diagonal matrix, and vote with precision matrix (subsection 4.4.3). 


In Scenario 2, each vehicle has an unknown precision matrix, which simulates 
e.g. different vibration characteristics. However, the combination of the 
predictions made by all vehicles is still performed with precision matrix used 
for scenario 1. The difference of the vehicle and global precision matrix 
could correspond to differences in the vehicle mass or various environment 
conditions and therefore a slightly different vibration behavior of the vehicles. 


For the simulation of the further scenarios the output fusion is performed only 
with the method vote with precision matrix since the results of the previous 
scenarios have demonstrated the highest accuracy with this method. 


In Scenario 3, one vehicle with a problem of the classifier is simulated, which 
is referred to vehicle with broken sensor. It classifies the segments totally 
random and the question is, how much impact it has on the final prediction. 
To decrease the negative impact of such a scenario, the following method is 
applied. 


After 5,000 overrun segments and at least 3 vehicles, the simulation compares 
the output of one vehicle with the output of the other vehicles, which are fused 
with the system precision matrix as explained in earlier scenarios. Lines 8 to 
16 in Algorithm 4.4 show the pseudo-code of this method, which is embedded 
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in the simulation algorithm. Based on these comparisons, the precision matrix 
ofthe one vehicle is estimated and checked if the values differ greatly from the 
back-end precision matrix. 


If the difference exceeds a value, which is set to 0.15 in this simulation, the 
vehicle with the equally distributed precision matrix and wrong predictions 
is identified. Furthermore, the estimated vehicle precision matrix is used for 
further calculations to reduce the confidence of the output of this vehicle and 
therefore reduce the overall number of misclassifications. 


For the simulation of this method and for Scenario 4, the number of segments 
is set to s = 10,000 and one run of the simulation per vehicle is performed. 
The properties of the segments, representing one of the five classes, are nearly 
uniformly distributed. 


In the simulation, the ground truth changes after 5 runs for every third road 
segment. It depicts situations like the fix of road damages or the formation 
of new ones over time. The scenario should clarify that the back-end needs 
a forgetfulness of previous classifications to guarantee a fast reaction to the 
change. As benchmark it is counted how many of the s = 10,000 segments are 
wrongly classified when using the vote with precision matrix without forgetting. 
Additionally, a modified version is implemented, which has the ability to forget. 
Therefore, the vehicle observations o), with k = 1,...,3, are scaled down. E.g. 
the classification of vehicle k from time step t — i has only the weight f’ in time 
step t with f € [0,1]. A forgetting factor of f = 0.5 is used for the simulation. 
A higher factor means less forgetting because the observations are weighted 
stronger. 
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Algorithmus 4.4 : Estimate and adjust precision matrix. 
Input : n is number of vehicles, s is number of road segments, 
ml) is precision matrix of vehicle k € 1,...,n, Ip is 
precision matrix stored in the back-end, / is number of 


classes, ¢ is threshold for using no) instead of IT, 
Output : number of misclassifications with adjusted precision 
matrix of vehicle k 


for v — 1 to n do 


1 
2 for i — 1 to s do 
3 for 1 to v do 
4 vehicles drive over road segment (ground truth) 
with II, 
5 combine predictions of all vehicles with II, 
6 count number of misclassifications from combined 
predictions 
7 if i mod 5,000 = 0 and v > 3 then 
8 combine predictions from vehicles 1...v with Hp 
without vehicle k from last 5,000 road segments 
9 estimate precision matrix 1) from the comparison 
of predictions of vehicle k with combined 
predictions 
(k)_ 
in d= zale Tp | 
11 if d > t then 
12 use nm for vehicle k instead of II, for further 
calculations 
13 display: ‘confusion matrix for vehicle k was 
adjusted, d, mf: 


4.5 Conclusion of novel methods for road 
condition estimation 


This chapters describes the developed and applied methods to process inertial 
and acoustic sensor data for road condition estimation. Firstly, preprocessing is 
applied to align the labels, the desired output, with the raw data. Furthermore, 
the data in time domain is transformed in space domain by velocity and time 
vector. The data is linearly resampled to a uniform sample rate of 100m=!. 
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The rate sensor data is smoothed with a median filter and the vehicle velocity 
and acceleration data with a Savitzky-Golay filter. 


Before presenting methods to process the data, influencing effects on the sensor 
data or the output variable are experimentally identified. Besides the road 
condition, the velocity is the main influencing effect of vehicle body vibration. 
The main parameter influencing the sound pressure in the tire cavity is also 
velocity besides the road roughness or obstacles. The type of tire, surface 
temperature, tire temperature, tire pressure, and load have minor impact on the 
sound pressure. 


Incorporating these effects, methods to estimate the road unevenness and 
roughness with non-labeled data are presented. The output of the road condi- 
tion estimation is a colored map. Moreover, obstacles are determined, high- 
lighted, and listed in a table with information, such as number of drivers, size, 
average velocity. Furthermore, a trend identification is developed based on 
regression of the road condition over time. For data, which are labeled with 
the desired output variable, methods are presented to predict road attributes 
based on inertial sensor data. Moreover, a regression of sound pressure level 
in the tire cavity and influencing factors, such as velocity, is introduced to 
estimate the road roughness. The classification process, possible parameter 
variations, and performance measures are exemplarily demonstrated and dis- 
cussed with a benchmark data set, which incorporates three classes and three 
features, representing the vehicle vibration. 


To combine the prediction from the classification of various vehicles, the 
following procedure is developed. Firstly, similar trajectories are identified 
with range search and distance algorithms to combine the classification results 
of only those trajectories. Secondly, methods to combine these classification 
results are presented, which are referred to as majority vote, vote with precision 
diagonal, and vote with precision matrix. 


Overall, the presented methods cover the whole data processing chain to esti- 
mate the road condition with vehicles as sensor platform, from preprocessing 
to visualization and multiple vehicle combination. 
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Outline: This chapter describes the developed measuring devices as well as the 
software to acquire, transmit, and centrally store the data (Section 5.2). The 
presented methods to estimate the road condition are implemented in a Matlab 
toolbox, which is introduced in Section 5.3. Overall, the measuring devices and 
the software to acquire and analyze the data are developed and implemented 
under the following aspects, also to allow the application of a large number 
of measuring devices: automation, low-cost, no manual intervention, only 
few different computing environments and programming languages to allow 
a continuous data processing flow. The software for the measuring device to 
acquire the data is written in Python with bash scripts for routines, the data 
processing methods are implemented in Matlab. 


5.1 Overview 


The vertical vehicle dynamics simulation to vary parameters and to investigate 
their influence on road condition estimation with data mining, described in 
Section 3.5 is implemented in Matlab. The corresponding code and description 
to perform the analysis can be found in [147]. 


The methods to combine the classifications derived by various vehicles of an 
identical road segment are also implemented in Matlab. The developed toolbox 
is published in [180]. 


The following sections focus on the implementation of data acquisition and 
a Matlab toolbox to process the acquired sensor data. The complexity of 
the selection and combination of hardware, especially the sensors and data 
logger in the vehicle and the acoustic sensor in the tire cavity, as well as 
the requirements of the measuring device are outlined in Section 3.4. This 
chapter mainly presents the final result and application of the implementations 
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and describes the most important steps of the set-up and operation of the 
measuring devices. The detailed instructions for setting up and operating 
the measuring device to acquire data can be found in [181]. The instruction 
includes the self-developed software and installation file, files to 3D-print the 
cases, files for the self-developed printed circuit board (PCB) and software for 
the microcontroller. The developed toolbox in Matlab, named Vehicle Learner 
Toolbox, consisting of the implementation of the data processing methods, 
presented in Section 4.1 and 4.2 is published in [182]. 


Overall, the measuring devices and the toolbox are developed to enable an 
automatic data acquisition as well as processing. For the set-up and instal- 
lation of the measuring devices, only few manual interventions are needed. 
The measurements run automatically and the toolbox only needs the desired 
parameters as input. 


5.2 Data acquisition 


The developed methods and described sensors for acquiring sensor data auto- 
matically (Section 3.4.3) are the basis for the measuring devices developed in 
this work. The measurement system for acquiring inertial sensor data, vehicle 
position, and velocity and to transmit the data to a central data base is shown 
in Figure 5.1. To annotate the raw data with the label of the ground truth, two 
buttons are installed. For every measurement drive, data describing a certain 
type of surface (e.g. if asphalt 1, otherwise 0) and an attribute (e.g. if pothole 
1, otherwise 0) are recorded. After the measurement drives, binary coding of 
the data of the respective files is transformed into the coding given in Table 5.1. 
The unix time t, ID for the sensor, speed vy, position and time stamp of the 
GPS lat, lon, and tgps, accelerations a and rotation rates w along all three 
axes, and the two labels for the attribute e and surface type m are recorded and 
stored in a csv-table on the measurement system (Figure 5.1). 


The sound pressure in the tire cavity is acquired with a self-developed telemetry 
system incorporating the novel methods described in Section 3.4.3, which is 
attached to the wheel (Figure 5.2). The audio data are transmitted via Bluetooth 
to the measurement system in the vehicle, along with the state of charge (SOC) 
of the battery, and saved as a wav-file. The tire pressure monitoring system 
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Figure 5.1: Measurement unit, consisting of Raspberry Pi, inertial sensor, Adafruit ultimate GPS 
Hat, UPS Hat, and two buttons for labeling the data. In service, the inertial sensor 
is fixed under the Raspberry Pi on the bottom of the case. The case is cut open for 
visualization purposes. 


Table 5.1: Label and name of the classes for road surface and road attribute. 


m - Road surface e - Road attribute 
0 - Unknown 0 - Unknown 
1 - Smooth surface 1 - Good 


2 - Damaged asphalt 2 - Light damages 

3 - Damaged concrete 3 - Pothole 

4 - Cobblestone 4 - Manhole cover 
5 - Railway crossing 
6 - Speed bump 


(TPMS) data is transmitted on the radio frequency 433 MHz and fused with 
the inertial sensor data among others in the csv-table. 


To enable an automatic installation of the self-developed software, the SD card 
for the Raspberry Pi needs to be written with a self-developed image analogous 
to anormal Raspian OS image. The basis as Raspian OS of the self-developed 
image is Jessies. Alternatively, a new self-developed image can be created 
with the script kernel_img_push-to-SDCard.sh on Linux. To download and 
install the software on the Raspberry Pi, it needs to be connected to the internet 
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(a) telemetry system without wheel (b) telemetry system attached to the wheel 


Figure 5.2: Telemetry system, consisting of an acoustic sensor and TPMS in the tire cavity, a 
self-developed y-valve for cable entry and air, and the wireless transmitter in a case 
(without cover for visualization purposes). 


via LAN cable to automatically download the self-developed software from a 
Git-repository on a web-based hosting service. For the installation process, 
there is no manual interaction needed. If the Raspberry Pi is to be coupled 
with the telemetry system and the software is to be installed on the Bluetooth 
module of the board, the Bluetooth antenna needs to be plugged in and the 
Raspberry Pi needs to be connected to the telemetry via UART. The credentials 
for the access point and the remote data storage to transmit the data can be 
adjusted in the repository in the file rpi_config.cfg in the sections ° WLAN’ and 
’ Samba’ or directly on the Raspberry Pi. 


The micro-controller of the telemetry system, which is responsible for ac- 
tivating the PCB by rotating vehicle wheels, needs to be flashed with the 
self-developed software. Furthermore, the firmware of the Bluetooth module 
on the board needs to be updated. The software of the Bluetooth module is 
automatically installed during the installation of the Raspbeery Pi. 


The software for the measurement system in the vehicle is mainly developed 
in Python, routines for automation are implemented as bash scripts. The 
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sensorrecorder.py is the main measurement collection service of the software. 
The collection service runs the processing task of every sensor in a separate 
thread, as illustrated in Figure 5.3. The threads are responsible to acquire 
data from global position system (GPS), inertial sensor referred to as IMU, 
TPMS, and on-board diagnostics (OBD) and are divided into initializing and 
running. On program start, the threads initialize the required libraries for the 
sensors, which are initialized and set up. For GPS the gps3 and for IMU the 
rtimu library are imported. Furthermore, the settings, such as orientation, and 
the poll interval are set for IMU. For TPMS and OBD, the serial connection 
to the receivers are started. Moreover, the pin configuration for the buttons 
is initialized. Afterwards, the sensor threads get data. In addition, the GPS 
thread generates data dictionary and the IMU thread log the sample rate. OBD 
and TPMS thread decode the received data. In this work, only the data for 
vehicle velocity and wheel revolutions per minute are acquired. However, it is 
possible to sample other sensor data from OBD. The button thread checks the 
states of the pins. While running, the service collects the data from all sensors, 
combines and tags them with a timestamp and queues the data for writing them 
to an output file. The write-out task then will write the measurement data from 
all tasks into a file, creates a new one after five minutes and compresses the 
old one. If a sensor or receiver is not connected to the measurement system, 
the data from the others are still acquired. 


The audiorecorder.py to acquire the audio data from the telemetry system at 
wheel runs separately from the main collection service. It starts automatically 
triggered by userspace /dev (udev), a device manager for the Linux kernel, as 
soon as a bluetooth connection between the Raspberry Pi and the telemetry 
system is established. The telemetry system is configured as master and 
actively tries to connect to the Raspberry Pi. If the connection is successful, 
the recorder starts the audio transmission via a remote console on the bluetooth 
module of the telemetry system and the audio data is acquired and saved as 
wav-file. 


New files are created every 5 minutes to avoid large data files for the upload to a 
remote file storage. The file names contain an ascending ID and the timestamp 
to allow merging the audio data with data from the other sensors. 


The wifiupload.py script is triggered by the uninterruptible power supply (UPS) 
module, which manages the power outages and shutdown of the measurement 
system. The script assumes, that the vehicle’s engine is powered off and a 
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GPS IMU TPMS OBD 
Init: Init: Init: Init: 
— gps3lib — rtimulib — start serial — start serial 
— set settings 
— set poll interval 
Run: Run: Run: Run: 
— get data — get data — get serial data — get serial data 
— generate — get data sample from TPMS from OBD 
data dictionary rate receiver receiver 
— decode TPMS — decode velocity 
data and rpm data 
i t i i 
QUEUE 
t | t 
Unix Timestamp Button 
csv-writer 
- Init: 
Init: — pin configuration 
— open file for writing data 
Run: 
Run: — check pin state 
— request queue for new data 
— write measurement data to the file 


— create new data file every 5 min 


Figure 5.3: Structure of the main da ta collection service sensorrecorder.py with processing 
tasks for each sensor as separate, independent threads. The data is combined and 
tagged with a timestamp in a csv-table. 


proper WIFI link is available. If these conditions are valid, the script will update 
the Git repository and transmit the files to a remote file storage. Afterward, 
it will switch off the measurement system or restart the recording software, if 
power is available again. 


During the data recording, one can connect to the measurement system via 
LAN to monitor the recording and, if necessary, to save the data files. On a 
Windows PC, the programs ’ Putty’ and ° WinSCP’ are suitable for this purpose. 
The IP address of the measurement system is set to 192.168.111.3 and the data 
is located in the folder /home/pi/logging data. 


Overall, the measurement device runs mainly autonomously. The measure- 
ments of the sensors in the vehicle body start and stop with the ignition of 
the vehicle and the measurements of the sensors at the wheel with its rotation. 
After the measurement is finished and the data logger is connected to a known 
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access point, all data is automatically transferred to server. From there, the 
data can be processed with the toolbox, which is presented in the following 
section. The only manual intervention needed is to charge the battery of the 
telemetry system at the wheel via mini USB with output voltage of 5V after 
more than 150 h of measurement time.. 


5.3 Data processing 


5.3.1 Overview and import 


To facilitate operation by non-experts, the methods presented in this work to 
preprocess and process the data (Section 4.1 and 4.2) are implemented in a 
graphical user interface called Vehicle Learner Toolbox. It is based on Matlab 
and implements several machine learning operations of the freely available 
toolbox SciXMiner [183] (formerly, Gait-CAD [184]). The Vehicle Learner 
Toolbox provides the possibility to 


e import vehicle sensor data in different file formats, 

e process the imported data and automatically extract various features, 
e train a classifier model with a wide-ranging set of options, 

e test the trained classifier with a test set, and 

e visualize the results with the help of plots and maps. 


A project folder can be selected and sensor data can be imported in the corre- 
sponding menu element Data (Figure 5.4). There is the option to assign the 
sensor data to specific vehicles, since they vary in suspensions, damping, and 
other parameters, which has an impact on the vibration behavior. Therefore, 
supervised learning data processing, described in subsection 5.3.3, can be per- 
formed for data dependent on the vehicle. The data from various vehicles can 
be merged for the unsupervised learning data processing. The import allows 
.csv and .xIsx file format with the following column headers: 


e timestamp (unix timestamp), 

e x-, y-, z-accel (the acceleration values in each direction), 

e x-, y-, Z-gyro (the gyroscope values in each direction), 

e gps-timestamp (format: YY YY-MM-DDThh:mm:ss,000Z), 
e lat, lon (position in latitude and longitude), 
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TE Vehicle Learner Toolbox e oO x 
Project Data Unsupervised Learning Supervised Learning = 
Import Data 
Change Windows Import Data 
Delete Window Profile | 
Reassign Data 
¢ v Add new vehicle 
Delete Data 
Choose folder: Folder picker 
Choose file format: Choose data type: 
O .csv CO unlabeled data 
© xlsx @ training data 
O .csv and .wav O test data 
Delete original files: 
window profile: standard profile PS 


Import Name: 


Import Data 


Figure 5.4: The import data frame of the Vehicle Learner Toolbox. Different vehicles can be 
chosen and the data type can be set. 


e speed (in m/s), and 
e m, e (surface type m and attribute e labeling, if the data is not labeled, 
these columns should only contain zeros). 


Furthermore, csv files along with wav files can be imported, iftire cavity sound 
pressure was acquired. The sensor data from the csv and wav files are aligned 
and merged based on the unix timestamp. 


The data is imported and preprocessed with the methods presented in Sec- 
tion 4.1. Moreover, the imported data set can be categorized as training, 
testing, or unlabeled data. 


Furthermore, the parameters for the length of road segments and overlapping 
ofthese segments as well as the resampling frequency can be determined. The 
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standard window profiles are surface type with a window length of 50 m and 
attribute with a window length of 5 m. 


5.3.2 Non-labeled data processing 


The sections of the toolbox for the non-labeled data processing for inertial 
sensor and acoustic sensor data automatically apply the methods presented in 
subsection 4.2.3. Figure 5.5 shows the frame of the toolbox for the processing 
of inertial sensor data. The frame for the process of acoustic data is similar 
except for the possibility to filter the data for the range of the tire pressure and 
tire temperature. The data of various vehicles can be combined and the time 
interval as well as the area of interest can be chosen. Furthermore, the lower 
threshold and upper threshold as well as the resolution for the algorithms needs 
to be selected. Furthermore, the maximum, mean or the standard deviation 
of the score for each road segment can be calculated. The output of this data 
processing can be a map based on open street map (OSM) format with the 
road state scores, a csv file with the most damaged road segments and an OSM 
colored with the trend of the road state score. 
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@ Vehicle Learner Toolbox am D x 
Project Data Unsupervised Learning Supervised Learning ` 
Tire Sound Plot Unsupervised Plot 
Choose vehicles Bau a 
v 
window profile: standard profile J 
Time interval ; 
1day 3 days 1 week 1 month 6 months 1 year 3 years all 
C pilot all data 
Latitude range: 48.915 ~ 48.934 
Longitude range 8.77 - 881 
Lower threshold: 08 Upper threshold: 09 
Resolution width 1000 Resolution height 4000 
Resi Coceiation Osu OMean © Max 
Output type normal plot 
Miwebmap 
M csv file 
F trend detection 
Plot data 


Figure 5.5: The non-labeled data processing frame of the toolbox. 
5.3.3 Labeled data processing 


After the import and preprocess of the data, new data streams are calculated 
and features are automatically extracted, as proposed in subsection 4.2.4. Any 
code to calculate new data series or features can be easily added in the corre- 
sponding Matlab function. The in Section 4.2 proposed data mining methods 
are implemented in the toolbox. The user can select and use the methods under 
the menu element Supervised Learning (Figure 5.6). Furthermore, the toolbox 
contains additional data mining methods from the SciXMiner toolbox, intro- 
duced in subsection 5.3.1, which the user can select to process data. However, 
this work concentrates on the data mining methods described in Section 4.2. 
These methods showed the best performance compared to other methods from 
SciXMiner, as investigated in [203]. 
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[E] Vehicle Learner Toolbox = o x 
Project Data Unsupervised Learning Supervised Learning ` 
Generate Training Data 


Sound Dato Main Classifier | 


Errea BMW Test Classifier Be 
Choose training data: waiikportt Plot Classified Data Br 
Choose classifier: all data classifier v Add classifier 
Selection of single features: MANOVA v 
Number of selected features: 10 
Preselection of features Downgrading of correlated single features 
Feature aggregation Discriminant analysis (DA) v 


Number of aggregated features: 2 


k-folded-crossvalidation: 5 

Choose classification method: SVM v 
Multi-class method: 4vs1 v 
Choose kernel: Feen v 
Kernel order: 1 

Penalty term [c]: 10 

Epsilon [z]: 1 


Start training! Plot the train data on a map 


Figure 5.6: The train classifier frame of the Vehicle Learner Toolbox. Various classifiers can be 
trained with a wide range of options. 


From the training or testing data set, labels are encoded according to Table 5.1 
and data annotated with specific labels can be excluded from the classification. 
Another option is to thin out classes with significant more data points than other 
classes to allow an approximately uniform distribution of data points among 
the classes to prevent over-fitting of specific classes. Furthermore, systematic 
errors during labeling the data can be removed, e.g. if the button to annotate 
the data live was activated too early or too late the annotation can be shifted or 
data points with the wrong annotation can be excluded. After the generation 
of the data set to be processed, the settings for the classifier can be determined 
under the tab Train Classifier, shown inFigure 5.6. 


A new classifier model can be created with a training data set or an existing 
model can be selected to test on new data. The next part contains the settings 
of feature selection (e.g. multivariate analysis of variances (MANOVA)) and 
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aggregation (e.g. discriminant analysis), as proposed in subsection 4.2.4. 
Reducing the amount of features highly influences the classification result by 
reducing the chances for over-fitting. Itis possible to cross-validate the training 
process by setting the k-fold-cross-validation value to higher than 1. The last 
section offers a variety of settings for the classifier, e.g. for a support vector 
machine (SVM), including the kernel function and penalty term. Afterwards, 
the classifier can be trained and data can be plotted on OSMs. Furthermore, 
the confusion matrix and the total loss is shown in the Matlab console. 


E Vehicle Learner Toolbox - o x | 


Project Data Unsupervised Learning Supervised Learning ~ 


Normal Data > raeeinn 


Choose classifier RegressionModel_20171117 Test Classitier 5 Add regression model 
Choose ehices ey Plot Classified Dats puma ~ 


v 


Window profile standard profile v 
Time interval ` 
1day 3 days 1 week 1 month 6 months 1 year 3 years all 
| plot all data 
Latitude range 89 - 9.065 
Longitude range: 83 - 18:54 
Generate data 
Select independent features: | spi a Select dependent feature: splScore v 
|! averageTireTemperature 
[ averageTirepressure 
v] speed 
L splScore v 


Reduce Dataset (0-1) 08 
Cross Validation 5 


start regression 


Figure 5.7: The train regression frame of the Vehicle Learner Toolbox. Various regressions can 
be trained with a wide range of options. 


Similar to training the classifier with features from inertial sensor data, a 
new regression model can be trained with various variations under the menu 
Supervised Learning and Train Regression (Figure 5.7). For example, the 
vehicle can be chosen and the time interval and area to be investigated can 
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be set. Furthermore, the independent variables of the regression as well as 
the dependent variable have to be determined. Lastly, the size of the dataset 
can be automatically reduced to avoid long calculation time and the number 
of folders for the cross-validation can be entered. The output is a plot with 
the regression function and the measured observations as well as performance 
measures. Analogous to the classification, new data can be tested as well as 
plotted on OSMs. 


For testing new data, a data set with modifications in time range and area to 
be analyzed can be generated as described for training, and a trained classifier 
needs to be selected. If the test data set is labeled, the output of the prediction is 
again a confusion matrix and the classification error. Moreover, the results can 
be visualized and plotted on OSMs. The trajectories will be cut into segments 
of different color referring to the corresponding classes, which are predicted. 


5.4 Conclusion of implementation 


This chapter demonstrates the developed measuring device to acquire data from 
inertial sensors, acoustic sensors in the tire cavity, GPS, OBD, and TPMS. The 
instructions for setting up and operating the measuring devices, including the 
software, technical drawings of the cases, the developed PCB, and a list of the 
required hardware, is available in [181]. The measuring device is designed 
for an automated installation and autonomous operation and therefore for mass 
production and comprehensive data acquisition with various vehicles. 


The methods to process the acquired data are implemented in a user friendly 
Matlab toolbox, which is available in [182]. With this toolbox, the data 
processing is automated, from importing and storing data to combination and 
visualization of the road condition estimates. The user only has to set the 
parameters for the data processing methods. 
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Outline: This chapter presents the results of the validation of the developed 
measurement device (Section 6.2) and the estimation of the road condition with 
non-labeled and labeled data from inertial and acoustic sensor (Section 6.3). 
The visualization of the road condition estimation is implemented in the devel- 
oped Matlab toolbox. Matlab provides maps based on open street map (OSM), 
on which the estimations can be visualized. These maps are only available 
colored and the investigation of the results works on monitors. However, for 
this work, the estimations are mapped with the open-source application QGIS, 
which provides Stamen’s toner maps as basemap layer that are black and white 
and more suitable to read. The results of the investigation of the influence 
of different parameters and vehicles on the classification accuracy in simula- 
tion are presented in Section 6.4. Furthermore, the methods to combine the 
prediction of various vehicles are evaluated in Section 6.5. Each presentation 
of results is followed by a discussion. In particular, the developed measuring 
device is discussed and contrasted with vehicle on-board sensors. Moreover, 
the results of the road condition estimation (subsection 6.3.3) and the imple- 
mentation of the presented methods on an electronic control unit (ECU) of 
the vehicle to run online (subsection 6.3.4) are discussed. Furthermore, the 
extended simulation approach is concluded and the application and results 
are discussed (subsection 6.4.3). Lastly, each section in this chapter includes 
open problems and suggestions for succeeding works with the topic of road 
condition estimation using vehicles as sensor platforms. 


6.1 Overview 


For the validation of the development measurement device, which is presented 
in Section 3.4, data were acquired with a BMW 116d driving specific maneu- 
vers. In addition to data from the developed measurement device, reference 
data from established measurement devices, which are too expensive for fleet 
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use, were simultaneously acquired. Based on the comparisons of the data, the 
developed measurement device is evaluated in Section 6.2. 


Data for road condition estimation were acquired over a period of three years 
with different vehicles, different drives and under different conditions, such as 
temperature and weather. The measurement device in the vehicle body with 
the GPS and inertial sensor was always placed between the driver and co-driver 
seat, glued on the armrest behind the hand brake. The acoustic sensor was 
attached to a free-rolling wheel. 


For the estimation of road unevenness and roughness with the methods pre- 
sented in subsection 4.2.3, the time consuming process of data labeling is not 
necessary. Therefore, data were constantly acquired over the period of three 
years, mainly with aFord Galaxy. In addition, aBMW 116d, VW Golf, Smart 
and Mercedes-Benz S-Class were used to acquire data to test the methods for 
different vehicles. Overall, over 300 h of data were gathered for the estimation 
of road unevenness and roughness with non-labeled data. Subsection 6.3.1 
shows results of the application of the methods from subsection 4.2.3 on the 
mentioned data set. Furthermore, the estimation accuracy is compared for two 
different vehicles on the same track and the influence of the combination of 
multiple estimation results is identified. 


To apply the supervised learning methods, explained in subsection 4.2.4, the 
data need to be manually annotated with the desired output variable, which is 
time consuming. Therefore, less data were acquired compared to the data set 
for non-labeled data processing. To ensure a generalization of the presented 
methods, measurement data were recorded on randomly selected roads in the 
Karlsruhe area. The velocity, the road conditions, and the environmental 
conditions of the data set vary in a wide range. In total, reference data are 
recorded over a distance of 250 km over a period of 6 months. The data were 
recorded on 14 days by three different drivers. The acquisition of reference 
data is a lengthy process, as individual attributes, such as potholes, need to 
be overrun several times with varying approach angles, vehicle lanes and 
velocities. The results of applying the methods from subsection 4.2.3 on the 
described labeled data set are shown in subsection 6.3.2. 


The variation of vehicle parameters and the influence of this variation is inves- 
tigated in the extended simulation approach, presented in Section 3.5. Vehicles 
used for real measurement data acquisition, are replicated in simulation to gen- 
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erate a larger data set. The results and the robustness of the application of the 
methods from subsection 4.2.3 on the data set generated with simulation and 
varied paramaters are presented in Section 6.4. 


The basis for the multiple vehicle combination, such as the precision matrix, 
is derived from real measurement data. However, the calculations to combine 
the classification results are performed offline and results and evaluation of the 
combination strategies are shown in Section 6.5. 


6.2 Verification of the developed measurement 
device 


Inertial sensor 


The performance of the microelectromechanical systems (MEMS) inertial sen- 
sor is compared to a dedicated inertial measuring unit IMU) (OXTS-RT2502 
from Oxford Technologies Soluations Ltd.), which is accurate but too expen- 
sive for fleet use. Figure 6.1 shows the comparison of the signal of the MEMS 
inertial sensor and from OXTS-RT2502, which is referred to as reference, 
for various maneuvers, including slalom, acceleration and deceleration, and 
a cleat test. The filtered signals are smoothed with a Savitzky-Golay filter 
with parameters k = 7 and f = 21 for the vertical acceleration and with k = 3 
and f =51 for others. Although the vertical acceleration show higher noise 
(0.1921 m/s”) compared to the reference sensor (0.1428 m/s”), Figure 6.1 in- 
dicates that the MEMS sensor is suitable to measure the vehicle motion with a 
similar accuracy like the dedicated data logger. Table 6.1 shows performance 
measures for signals of the maneuvers and proves this suggestion. 


Acoustic sensor 


Figure 6.2a exemplarily shows the power spectrum from the CMA-4544pf-4w 
std. from the THD test, which is the basis to calculate the THD value of the 
sensor. Figure 6.2b shows the SPL for the microphone INMP411 from the 
excitation of the tire with an impulse hammer. The maximum SPL in the tire 
cavity was 165 dB. However, the microphone INMP411 shows a plateau with 
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Figure 6.1: Comparison of signals from MEMS and reference inertial sensor. 


Table 6.1: Summary of statistics for reference and MEMS inertial sensor signal. 


Statistics 

Maneuver 

R? NRMSE 
Start and braking 
Longitudinal acc. (unfiltered) 0.99 2.09 
Longitudinal acc. (filtered) 1.00 0.53 
Slalom 
Lateral acc. (unfiltered) 0.99 2.35 
Lateral acc. (filtered) 1.00 0.86 
Yaw rate (unfiltered) 1.00 0:55 
Yaw rate (filtered) 1.00 0.22 
Cleat test 
Vertical acc. (unfiltered) 0.84 4.10 
Vertical acc. (filtered) 0.93 2.52 


a SPL of 132dB, which corresponds to the acoustic overload point of 131 dB 
specified by the manufacturer. 
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Figure 6.2: Results from THD and SPL test. 


Figure 6.3 shows the frequency response of all microphones during vehicle 
operation. The figure indicates that the frequency response of the modified 
conventional electret microphone (CMA-4544pf-w mod.) approximates the 
response from the reference sensor (PCB 103B02) best. 


— PCB 103B02 —— INMP411 ——— CMA-4544pf-w std. —— CMA-4544pf-w mod. 
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Figure 6.3: SPL with a 1/12 octave band smoothing of all microphones during vehicle operation 
with constant speed of 60 km/h. 


Table 6.2 lists quantitative criteria of all sensors and tests. The table under- 
lines, that the modified electret microphone measures almost the same sound 
pressure level (SPL) as the reference microphone. Furthermore, it has a sim- 
ilar total harmonic distortion (THD) value and power spectral density (PSD) 
values for the test runs with 30 and 60 km/h compared to the reference micro- 
phone. Lastly, R? between the reference and each microphone is highest for 
the modified electret microphone. 
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The results of the experiments suggest, that the microphone CMA4544PF-w a 
modified configuration is a good alternative to the expensive PCB 103B02 to 
perform acoustic measurements of the tire cavity sound. 


Table 6.2: Summary of acoustic and statistics of the experimental examinations. 


Measures 

Max. THD Max. Max. 
Microphone input (110 dB, PSD, PSD, R2 

SPL 1 kHz) 30km/h 60km/h 

(dB) (dB) (dB) (dB) 
PCB 103B02 > 165 -65.9 25.1 32.6 - 
INMP411 132 -50.7 19.7 24.4 0.77 
CMA4544PF-w std. 140 -37.0 22.0 26.4 0.83 
CMA4544PF-w mod. 161 -63.4 25.20 33.1 0.96 


GPS 


The accuracy of the GPS was determined by stopping a test vehicle five times 
at the same position. The orthodromic distance, the shortest distance between 
two points in the surface of a sphere, from the five acquired positions is 
illustrated in Figure 6.4a and the standard deviation of the distances is 1.12 m. 
Figure 6.4b shows the comparison of the vehicle velocity obtained from on- 
board diagnostics (OBD) and global position system (GPS). The signals have 
a similar graph, the R? is 1.00 and NRMSE is 1.90. Overall, the GPS sensor 
has a good accuracy in velocity as well as in position. 


Discussion 


A novel measurement device for acquiring GPS, inertial sensor, and acoustic 
sensor data is presented in this work. It works autonomously and any vehicle 
can be equipped with this device. 


However, inertial sensors are even partly integrated in the standard equipment 
of new vehicles for fusion with GPS for a more precise position determination. 
The data from the measuring device used in this work are compared with 
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Figure 6.4: Orthodromic distance from five GPS signals at the same position and velocity from 
GPS and OBD . 


inertial sensor data from an Audi A7. The sliding standard deviation of the 
rolling rate, pitch rate and vertical acceleration of both sensors proved to be 
almost identical with coefficient of determination of 0.9577 on average and 
a normalized root mean square error (NRMSE) of 0.0416 on average, which 
we have showed in [185]. Therefore, the developed methods for the inertial 
sensor can be applied on modern vehicles without the employment of an 
external measuring device. The advantage of the inertial sensor is that it is 
inexpensive and already on-board in many new vehicles. In comparison to 
a camera, inertial sensors can only be used to evaluate the directly traversed 
lane area. However, data of the inertial sensor already contain information 
about minor unevenness which causes the vehicle to vibrate, which might be 
mistaken by the camera. Moreover, road condition estimation with cameras is 
problematic under varying conditions, such as darkness and moist on the road 
surface. All in all, a system consisting of inertial sensor is a cost-effective and 
computationally efficient alternative to cameras to monitor the condition of the 
road infrastructure. The fusion of the two systems represents a possibility to 
optimize mobile condition estimation. 
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6.3 Road condition estimation 


6.3.1 Road unevenness estimation and trend identification 
with non-labeled data 


Figure 6.5 shows the estimation of the road unevenness index (RUI) based on 
the data processing methods presented in subsection 4.2.3. The frame length 
for the road unevenness index (RUI) estimation is 5m. The road condition 
estimation for the highway B/0 in Niefern, Germany, in the figure are based on 
inertial sensor data acquired with a VW Golf. The estimation shows areas with 
severe unevenness, colored red. These locations are automatically detected and 
tagged with a circle with the double thresholding and connected-component 
labeling algorithms (subsection 4.2.3). The tags are shown in Figure 6.5 and 
labeled with number 1 to 4. The estimations are compared to the officially 
determined condition assessment and evaluation 2015 (ZEB). 
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nous road uneveness estimation in Niefern, Germany. 


Label 1 and 2 are bridge gaps, which have a strong excitation of vibration of 
the vehicle, and Label 3 and 4 are severe road surface defects. Overall, the 
coefficient of determination for the estimated RUI compared to the ground 
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truth for the whole test drive around Niefern is 54% and for the unweighted 
aggregation of various drives 63%, which we have showed in [185]. The 
coefficient of determination for a different vehicle, a BMW 116d, for the 
same test track is 50% [185]. The coefficient of determinations show, that 
the estimation can explain a large portion of the ground truth and that the 
aggregation of various drives improves the estimation result. Furthermore, the 
estimation seems to be stable for different types of vehicles. The R? of RUI 
estimation of the autobahn (A6) close to Niefern with the VW Golf yields 
45 %, which corresponds to the estimation accuracies for highway B10. The 
estimation accuracy based on a camera is higher for B10 with 76 % and similar 
for A6 with 47 %[185]. 


Table 6.3: Automatically detected hotspots with severe unevenness. 


max(RUI) (RUD long. lat. f u(velocity) or(velocity) drives 
(m) (m/s) (m/s) 
1.00 0.71 8.77 4892 1.6 13.59 3.38 11 
0.97 0.73 8.44 48.98 23 22.32 0.04 2 
0.83 0.69 8.65 48.91 1.0 24.29 0.04 4 
0.75 0.68 8.65 48.91 1.0 23.61 0.04 4 


Table 6.3 exemplarily shows the output of the toolbox with the automatically 
detected severe unevenness hotspots. In addition to maximum and on aver- 
age estimated RUI, max(RUI) and (RU), of the corresponding hotspot, the 
coordinates of the midpoint of the hotspot, long. and lat., are displayed. Fur- 
thermore, the radius of the hotspots r and the average and standard deviation of 
the velocity of vehicles driven over the hotspots, j(velocity) and ø (velocity), 
and the amount of drives are automatically calculated. 


Figure 6.6 exemplarily shows the estimation of the RUI for a greater road 
network based on data acquired with Ford Galaxy between 2016 and 2017. 
The Labels 1 to 3 show road segments with severe unevenness, which are 
confirmed by the driver. 


The former examples show the advantage of the developed methods and tool- 
box. The data are automatically acquired, processed, and visualized. The 
maps with the colored estimation of the road condition and the table with 
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hotspots give a comprehensive overview of the road condition. However, the 
output is only an unevenness index, without knowledge of the actual attribute 
on the road. The results of the prediction of such attributes are shown in 
subsection 6.3.2. 
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Figure 6.6: Continous road uneveness estimation in Karlsruhe, Germany. 


Figure 6.7 shows the results of the trend identification method for RUI for 
Karlsruhe, Germany, and surrounding area. The trend identification method, 
which is presented in Subsection 11, estimates if the condition of a road 
segment decreased or increased over time. The figure indicates, that the quality 
of the road infrastructure remained constant except for small road segments 
in the city and larger areas marked with Label 1 and 2. With such an output, 
road infrastructure engineers can identify road segments, where the condition 
greatly decreases in short time intervals to identify the reason and to take 
countermeasures. The reasons for such decreases in road condition might be 
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a high portion of heavy goods vehicles or mistakes during the construction of 
the road. 


6.3.2 Road attributes and surface estimation with labeled 
data 


This work distinguishes between the classification of attributes and road sur- 
faces. The classification of road surfaces aims to give a rough overview of the 
condition of the road network. Hereby, the length of each road segment to be 
classified is longer, 50 to 100 m, compared to the segment length for attribute 
classification, where it is 5 to 10 m long. 
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Road surface classification separates the classes smooth asphalt and concrete, 
damaged asphalt, damaged concrete, cobblestones/off-road. Road attribute 
classification incorporates the classes potholes, manhole covers, railways 
crossings, speed bumps, light damages, good condition. 


Figure 6.8 illustrates the classification results of both approaches in aggregated 
features space. Table 6.4 shows the performance measures (subsection 2.5.6) 
of the classification. 
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Figure 6.8: Classification results with two aggregated features and borders of the classifier in 
black. 


The results, including Figure 6.8, and visualized examples for both examples 
are explained in more detail in Subsection 6.3.2, as well as the regression and 
visualization of the roughness estimation of road surfaces based on sound data 
of tire cavity. 


Attribute classification 


The cross-validated accuracy of classifying attributes yields 81 % on average 
without feature aggregation. The most important features to separate the 
classes, determined with multivariate analysis of variances (MANOVA) are: 
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Table 6.4: Results of single features classification. 


Precision Recall i Precision Recall 
surface type Attribute 

(Te) (1) (%) (%) 
smooth surf. 95.8 97.6 good 83.9 92.4 
damag. asph. 97.3 94.9 light dam. 78.5 70.8 
damag. con. 95.6 92.9 pothole 85.3 77.4 
cobblestones 99.1 100.0 manhole 40.9 29.1 
railway cro. 61.6 57.0 
speed bump 94.9 91.0 
Average 96.4 96.4 74.2 69.6 

Accuracy 96.1 81.0 


e peak-to-peak pitch acceleration 

e peak-to-peak roll acceleration 

e maximum of jerk in vertical direction 

e average root mean square (RMS) of the vertical acceleration 
e average velocity 


Previous analysis have shown that the vehicle velocity has a high influence 
on the sensor data and is therefore selected as an important feature for the 
classification. The aggregated feature space, aggregated with linear discrimi- 
nant analysis, and the lines of the function to classify the attributes are shown 
in Figure 6.8b. 


The illustration of the classification shows, that road segments in good con- 
dition, with light damages, speed bumps, and potholes can be separated well. 
This indication is proofed by the quantitative results, listed in Table 6.4. The 
precision and recall for good condition, light damages, speed bumps, and pot- 
holes is above 70 %, whereas the performance measures for manhole cover 
and railway crossing is below 62 % on average. 


By comparing each class with each other, it emerges that the peak-to-peak val- 
ues of pitch and roll acceleration are mainly responsible to separate attributes, 
which occur on 


e both vehicle lanes (railway crossing, speed bump), 

e on only one side of the vehicle (manhole cover, pothole), 

e or have only little impact on the vehicle vibration (light damages, road 
segments in good condition). 
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In addition, the average RMS of the vertical acceleration is important to sepa- 
rate light damages and road segments in good condition. Furthermore, potholes 
and manhole covers are dividable by the maximum RMS of the roll accelera- 
tion in the frequency range 15 to 25 m~!. However, latter attributes are often 
misclassified as segments in good condition or light damages. Speed bumps 
and railways crossings are separable by the value of the peek-to-peek of the 
pitch rate, whereas railways crossings are also often misclassified as light 
damages. 
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Figure 6.9: Attribute classification results of two high speed roads in the south west of Karl- 
sruhe, Germany. 


To test the classifier, a data set of more than 200 km of real data is classified and 
plotted on maps based on open street map (OSM). The results are promising 
and represent the actual street condition in many occasions. A few examples 
of classified areas are shown below. 


The first example shows the attribute classification results on two different high 
speed roads (Figure 6.9). The upper one with Label 1 is a freshly renovated 
asphalt highway with close to no damages and the lower one with Label 2 is 
a poorly patched asphalt road with a lot of medium and severe damages. The 
classification successfully predicted the upper roadway as good street. Most 
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parts of the lower street were predicted as light damage and some points even 
as potholes. The results conform to the actual road condition. 
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The second example presents data acquired in an urban area in Karlsruhe, 
the predictions are shown in Figure 6.10. The roads in this area are poorly 
preserved and there is a speed bump at a pedestrian crossing (Label 1). The 
classification model correctly predicts the speed bump (Label 1) for all overruns 
and a pothole (Label 2). 


The third interesting sector is shown in Figure 6.11. Potholes (Label 2 and 3), 
which were at the edge of the driving line, were overrun multiple times and the 
classifier predicts the severe damage accordingly. Sometimes the output at the 
road segments is not pothole but light damages or even good road condition. 
The reason might be that the pothole was avoided by the driver. 
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Figure 6.11: Attribute classification results of road segments outside of Karlsruhe, Germany, for 
the events potholes and railways crossing. 


The railway crossing (Label 1) is more elevated than other crossings and 
misclassified as speed bump in few cases. 


Road surface classification 


When classifying road surfaces, cross-validation yields 96.1 % accuracy on 
average without aggregation of features. The three best individual features for 
the classification of road surface according to MANOVA are: 


e RMS of the roll acceleration for frequency range from 5 to 15 m! on 
average 


e standard deviation of the pitch rate 


e stand deviation of the RMS of the vertical acceleration for frequency 
range 15 to 25 m=! 


The aggregated feature space is shown in Figure 6.8a. The figure indicates, 
that the biggest portion of misclassifications results from asphalt classified as 
damaged asphalt or damaged concrete and vice versa. The illustrated results 
are underlined by Table 6.4, where the precision and recall for cobblestone 
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is above 99.0 %, whereas the performance measures for asphalt, damaged 
asphalt, and damaged concrete are between 92.0 and 97.6 %o. 


The values of RMS of the roll acceleration and vertical acceleration separate 
the classes smooth surfaces, damaged asphalt and cobblestone. The values 
are highest for cobblestone and low for smooth surface. Standard deviation of 
pitch rate separates the classes damaged concrete from all other classes. The 
reason is probably poor and aged concrete joints. 


The surface classifier was applied on the same data set described for attribute 
classification. The classifier was able to reflect the road surface precisely. The 
following figures display the performance on different surfaces. Analogous 
to the attribute classification shown in Figure 6.9, the surface classifier could 
distinguish between both pavements and correctly classified them as smooth 
surface and damaged asphalt, respectively. 
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Figure 6.12: Road surface classification of road segments in urban area. 


The classification results from data acquired in the urban area of Karlsruhe 
(Figure 6.12) show two correctly predicted areas of cobblestone (Label 1 and 
2). The remaining road segments are correctly classified as segments with 
light damages or in good condition. Especially the latter class was correctly 
predicted for a road segment, which was recently renewed (Label 3). One 
misclassification of cobblestone can be found close to Label 1. However, this 
road segment is highly damaged with various potholes, which have a high 
impact on the vehicle vibration similar to cobblestone. 
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Figure 6.13 shows highway segments with aged concrete and distinctive con- 
crete joints (Label | and 2), which have to be maintained shortly. Except for 
one short segment, which was classified as smooth surface, the road state was 
correctly predicted. 
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Figure 6.13: Road surface classification of Karlsruhe, Germany, and highways with different 
surface types. 
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Road roughness estimation 


e observations A. regression function 
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Figure 6.14: Estimated regression function for road roughness index (RRI). 


Figure 6.14 shows the observations and estimated regression function for a por- 
tion of the acquired data. The regression error, expressed as normalized mean 
square error, of a five fold cross-validation of all acquired data is 4.54% on 
average and the standard deviation is 0.15 %0.The figure and the calculated 
error indicate that the regression function is able to reproduce the actual ob- 
servations very well and it can also be used for the estimation of road surface 
roughness for future measurements. 


For example, Figure 6.15 and 6.16 show the estimated roughness based on 
acoustic data acquired with BMW 116d with varied velocities. Figure 6.15 
shows the roughness on a highway from Karlsruhe to Niefern, Germany. 
Changes of road roughness (Label | and 2) are visible along the highway, 
which correspond to the actual roughness condition. 


Figure 6.16 indicates, that the roughness at the crossover in Niefern (Label 1), 
at which traffic is very heavy, is coarser compared to other road segments in 
Niefern, Germany. Furthermore, the highway leading to Niefern (Label 2) is 
new, which is represented by the results of this analysis. 
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Figure 6.16: Road roughness estimation in Niefern, Germany. 


6.3.3 Discussion of road condition estimation 


The results (Section 6.3) of the non-labeled data processing of the vehicle body 
vibration and the tire vibration give a good estimation of the degree of the road 
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unevenness and road roughness. The achieved coefficient of determination for 
two different vehicles is even comparable to results from recent studies, but 
those involve a training process, e.g. [90]. The output of the data processing, 
e.g. the maps or tables with hotspots, provide a comprehensive overview of 
the condition of the road infrastructure. Road inspectors can identify road 
segments in poor conditions and can structure and prioritize the maintenance 
of the infrastructure. 


In contrast to supervised learning methods, a costly labeling process for this 
approach is not needed. However, the method can not differentiate, if the 
vibration of the vehicle or the tire originate from road defects or from structural 
obstacles. But if the vibration of the vehicle is high and the RUI accordingly 
close to 1 due to astriking structural obstacle, a maintenance might be necessary 
even it is not a defect. Examples are railway crossings or bridge expansion 
joints, which are corrupted. 


Moreover, the thresholds for the algorithm to identify road segments with se- 
vere excitation on the vehicle are adjustable based on the purpose of the road 
inspector. The estimator of the RUI is dependent on the vehicle, since dif- 
ferent vehicles have various suspension systems and vibration characteristics. 
However, the estimates can be merged across different vehicles and trajectories. 


An extension or specification dependent on the vehicle of the feature set might 
improve the estimator. For example, the features based on the pitch rate can 
be considered. Moreover, the lower and higher frequency thresholds for the 
calculation of the RMS with can be specified for vehicles to decrease the 
portion of frequencies not relevant of road unevenness. However, this requires 
an extensive study of the vehicle vibration characteristic due to obstacles and 
a defined test track with various defects. 


The results of the data processing for the inertial sensor data show that both, 
road materials and attributes can be classified. The features automatically 
selected by MANOVA are in agreement with the theory of vehicle excitation. 
Surface type classification performs well according to the results of the cross- 
validation and the test data. 


There are various misclassifications among the prediction of attributes, espe- 
cially for structural obstacles, such as manhole covers and railway crossing. 
However, these attributes might be marked on a map and excluded from clas- 
sification and investigation, as the main objective is to detect road damage. 
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One reason for misclassifications of the attributes good condition, light dam- 
ages, and potholes might be false manual annotating, since there is sometimes 
only a fine line between the degree of damages, or the events were not fully 
overrun, especially for potholes. 


The inertial sensor and the developed data processing flow can be adapted 
to detect other road features, which might be interesting for transportation 
authorities. Road features, which might be classified with a good accuracy, are 
wavelike deformations in longitudinal direction. 


Other road infrastructure information, such as road slope or radius of curves, 
can also be estimated solely based on inertial sensor data, as we showed in 
[112, 186]. For this purpose, features from other time series, such as yaw rate, 
longitudinal and lateral acceleration of the vehicle, are more important than 
time series used for the estimation of road condition. 


The data processing of acoustic sensor data can be extended to estimate the 
tire road noise. In general, the degree of road roughness is positive correlated 
with tire road noise. However, there are exceptions, such as a open-pored road 
surface, which absorbs sound and reduces the tire road noise. Moreover, the 
microphone inside the tire cavity might also be useful to detect extended water 
residues on the road surface. 


Overall, the methods can be tuned and the results can be improved with novel 
demonstration, examination, and reference areas, such as duraBASt. Ground 
truth data for different surface textures for the estimation of the RRI and of 
different longitudinal unevenness for the estimation of RUI can be obtained 
with lower effort under controlled conditions. Moreover, ground truth data for 
defined road defects, which can be modified over time, can be acquired under 
defined environment conditions, such as rain and temperature. 


6.3.4 Discussion of online estimation 


The data processing presented in this work can probably run on electronic 
control units (ECUs) of modern vehicles. However, the ECU has limits in 
computational resources, such as working memory. The results of MANOVA 
of the labeled inertial sensor data processing show that the most important 
features are peak-to-peak or standard deviation of specific space series data. 
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The major task is to avoid the calculation of features for windows, since it 
is costly to keep all values. There exist online algorithms, especially for the 
calculation of mean and variances [187, 188], which can be applied to run 
the data processing on ECUs. Algorithms for the online calculation of other 
features such as peaks, can be adjusted from ECG signal analysis, e.g. [189]. 
Furthermore, there exist approaches for real-time frequency and harmonic 
evaluation, e.g. [190, 191]. 


6.4 Influences on classification performance 


6.4.1 Variation of vehicle parameters 


The classifier is trained with data, which is simulated from the BMW 116d 
passive full car model driving on a4 km-long road with fixed size of attributes. 
Figure 6.17 shows the classes in feature space, described by two aggregated 
features from 15 single features. The class pothole has two clusters because 
two different sizes of potholes, each with smaller variations in dimensions, are 
simulated, which have different impacts on the vehicle vibration. The figure 
indicates, that the road features pothole and cobblestone are clearly separable 
to the other classes. Unevenness is close to the road features railway crossing 
and cobblestone but still good dividable. However, railway crossing, manhole 
cover and smooth surface are very close and show some misclassifications. The 
reason is that the vehicle vibrations due to these road features in simulation are 
similar. 


The results of testing the classifier for different types of vehicles and vehicle 
set-ups are shown in Table 6.5. Overall, the generalization error remain good 
with a maximum of 10.2 % for S-Class. 


The classifier from BMW 116d is used to predict unseen data, which are 
simulated from models of different vehicles, loads and suspensions driving 
over a 2 km-long road with random size of attributes. The testing results are 
shown in Table 6.6. It is obvious that the classifier performs good with the 
same vehicle model. However, the accuracy of the classifier declines from 90% 
to 81% with the increasing load on the vehicle. For other vehicles and set-ups, 
the performance measures decline greatly, which means the mass, size and 


147 


6 Results and discussion 


e pothole e manhole e cobbled 
e railway e unevenness e asphalt 
| | | 
1 = > — 
& > 
o 
E 
is] 
2 
3 0.5 F + 
i) 
Bn 
© 
bb 
BD 
S 
[0] S = 
| | | 
0 0.5 1 


aggregated feature 1 


Figure 6.17: Classes separated by two aggregated features from 15 single features. 


Table 6.5: Performance of classifiers for different vehicles. 


Vehicle Precision Recall Accuracy ae 
error 
(%) (%) (%) (%) 
BWM 116d testing 90.4 90.8 90.6 7.6 
S-Class W220 testing 86.1 89.0 87.6 10.2 
Sprinter testing 94.9 94.9 94.9 5.0 
BMW 116d act. susp. testing 90.8 91.6 91.2 8.5 


parameters of spring and damping have an individual effect on the accuracy 
of the classifier. The results indicate, that the classifier, which is trained with 
BMW 116d, is not robust and the performance for the application on other 
vehicle set-ups greatly declines. 


The performance measures improve, ifthe original classifier is trained incorpo- 
rating the parameters and set-ups of the vehicles as features. Therefore, three 
vehicle model variations are considered as features, the load of the vehicle, the 
model of the vehicle and the type of suspension. The new classifier is trained 
on a road of length 180 km with a wide variation of vehicle loads, models, 
and type of suspension. Analogous to above, this classifier is tested on the 
presented vehicle models. The results (Table 6.7) suggest that this classifier is 
more robust against different vehicle models compared to the classifier solely 
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Table 6.6: Performance of the classifier for BMW 116d applied on different vehicle set-ups and 


vehicles. 
Vehicle Precision Recall Accuracy u 
error 
(%) (%) (%) (%) 
BMW 116d 200 kg loaded 87.1 88.3 87.7 10.5 
BMW 116d 400 kg loaded 81.2 80.8 81.0 17.2 
S-Class W220 77.2 68.4 72.8 25.4 
Sprinter 75.6 68.5 72.1 26.1 
BMW 116d act. susp. 42.7 38.0 40.4 57.8 


based on data from one specific vehicle. However, the performance measures 
for slightly changes of the set-up, e.g. the increase of load, decrease com- 
pared to results shown in Table 6.6. Overall, the lowest value for performance 
measures is 73.0 and the highest value for generalization error is 24.5 %. 


Table 6.7: Performance of the classifier for various vehicle models applied on different vehicle 
set-ups and vehicles. 


Vehicle Precision Recall Accuracy General, 
error 
(%) (%) (%) (%) 
BMW 116d 200 kg loaded 73.5 75.9 74.7 24.5 
BMW 116d 400 kg loaded 82.2 79.4 80.8 18.4 
BMW 116d act. susp. 80.4 78.0 79.2 20.0 
S-Class W220 81.4 74.0 TET 21.5 
Sprinter 73.8 76.2 75.0 22.2 


6.4.2 Variation of classification parameters 


The influence of the position of the inertial sensor in the vehicle body in the 
simulation is shown in Figure 6.18. The best accuracy can be achieved with 


output obtaining from position S2, namely in the middle above the front or rear 
axle. 


Moreover, the number of selected features and the order of the polynomial 
kernel also have an effect on the classification. In the practical application 
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Figure 6.18: Influence of the sensor position on the accuracy of the classification. 


on the vehicle, the classifier should at least maintain at a high level no matter 
how the load of the vehicle changes. Thus each data flow contains the output 
simulated by a selected vehicle model driving across different road attributes 
with a series of loads from 0 kg to 400 kg. The features are selected with 
MANOVA and the number of features is set from 10 to 30. The classification 
is performed with support vector machine (SVM) with a polynomial kernel with 
orders 1 to 5. The results, respectively for the BMW 116d with anti-roll bar, 
are shown in Table 6.8. The results suggest that the maximum accuracy can be 
achieved with a kernel order of 2 and 25 road features, which leads to 13.85 % 
misclassification. Using 25 features and kernel order 3 is the second best 
option with 14.38 % misclassification. The results indicate that the classes are 
better separable with nonlinear classification functions. Furthermore, a higher 
amount of features, e.g. 30, seems to lead to over-fitting. 


Table 6.8: Misclassifications in % of classifier with varying kernel order and number of features. 


number of selected features 
kernel order 10 15 20 25 30 
23.00 19.77 21.31 15.62 19.38 
21.92 1823 17.54 | 1385 17.46 
22.77 1762 1662 1438 15.92 
23.08 18.08 16.62 16.08 16.62 
23.31 1854 1772 1723 17.15 


run 
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6.4.3 Discussion of extended simulation approach 


The proposed extended simulation approach enables to investigate the influence 
of parameter variation of vehicles on the classification performance. The 
simulation consists of a full car model with 10 degrees of freedom (DOF), 
various extensions, such as active suspension, a modified point contact tire 
model, dependent on vehicle velocity and length of obstacles, and road models 
including road features and defects. Itis shown that it matches actual measured 
data, and therefore is a good representation of the real-world. Supervised 
machine learning techniques involve a costly and laboriously collection of 
labeled data for training and testing. The simulation reduces these costs, and it 
can be used to investigate and quantify the impact of specific vehicle parameters 
and settings on the classification results. Furthermore, the best parameters for 
the data processing and classification can be determined or tested. 


Further improvements are to incorporate additional elements of a real vehicle 
into the vehicle model to the point of a commercial complex simulation, such 
as CarMaker [192]. With the exact set up and behavior of real vehicles in 
the simulation, classifiers can be trained in simulation before assigning the 
classifier to real vehicles. 


6.5 Multiple vehicle combination 


The performance results of the proposed combination strategies for different 
scenarios, which are described in subsection 4.4.3, are shown and discussed 
in this section. Figure 6.19a shows the difference between the estimators for 
Scenario 1, where the precision matrix for the combination strategy and for 
all vehicles is identical. The vote with precision matrix, which incorporates 
the precision derived from the confusion matrix for all classes, is the one 
with fewest wrong predictions. Already with one vehicle, the proportion of 
wrong predictions is below 20 %, which means that each segment is classified 
correctly on average. With increasing number of vehicles, the number of 
wrong predictions for vote with precision matrix decreases faster compared to 
majority vote or diagonal vote. Between latter two estimates no big difference 
can be identified. Figure 6.19b represents the results of the prediction based 
on the vote with precision matrix split into the classes. The result shows a 
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high impact of the class 5, Cs, on the number of wrong predictions because 
the precision of the vehicles to predict class 5, 755, is only 0.73 and lower 
compared to other classes (Table 4.11). 
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Figure 6.19: Results of vehicle combination: Scenario 1. Each vehicle predicts the ground truth 
based on the global precision matrix. 


In Scenario 2, each vehicle has an unknown precision matrix but the combi- 
nation of the classifications is still performed with the global precision matrix 
(Table 4.11). As shown in Figure 6.20, again vote with precision matrix is the 
most promising estimate. When comparing one vehicle, which drives multiple 
times over the road and classifies the segments (Figure 6.20a), and multiple 
vehicles (Figure 6.20b), the latter one achieves better results. The reason is 
that the fixed precision matrix of only one vehicle can not compensate the 
difference to the global one and the proportion of wrong predictions converges 
towards 10%. However, five or more different vehicles perform better and 
surpass 10 % (Figure 6.20b). 


Figure 6.21 shows the influence of a vehicle with broken sensor from Sce- 
nario 3 on the number of wrong predictions. In Figure 6.21a, the first vehicle 
has or has not an equally distributed precision matrix and therefore randomly 
predicts the classes. Above five vehicles, in average less than 10 % of the 
road is classified wrongly. Above ten vehicles the impact of the bad precision 
matrix can be ignored. Figure 6.21b shows the results of a simulation, in which 
the third vehicle has an equally distributed precision matrix, which is referred 
to as broken sensor. The blue line shows the proportion of misclassifications 
with the third vehicle with a broken sensor and no adjustments. As soon as the 
output of the third vehicle is included in the simulation, the number of wrong 
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Figure 6.20: Results of vehicle combination: Scenario 2. Unknown distribution of classification 
due to different precision matrices for vehicles. 


classifications increases greatly. With increasing the number of vehicles with 
good precision matrices, the number of wrong predictions converges again to 
0. The green line shows the results of the simulation with the identification 
of the vehicle with the bad precision matrix and the adjustment of the preci- 
sion matrix in the back-end for this vehicle with the proposed Algorithm 4.4 
in subsection 4.4.3. With this adjustment, the increase of number of wrong 
predictions is not as high as without adjustment and the number of misclassi- 
fications faster converges to 0. Summarized, this method improves the results 


of our multiple vehicle fusion and can identify vehicles with bad classification 
outputs. 
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Figure 6.21: Results of vehicle combination: Scenario 3. Classification with and without broken 
sensor. a: lst vehicle with and without broken sensor. b: 3rd vehicle with broken 
sensor with and without automated adjusting precision matrix for this vehicle. 


The results of Scenario 4, where the ground truth changes at one time sample, 
can be seen in Figure 6.22. After the change of the ground truth at time sample 
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t=5, the vote with precision matrix without forgetting needs 4 more time 
stamps to reduce the proportion of wrong classified segments below 10 %. 
When using the forget factor, already in the next time stamp an error rate below 
2 % can be reached. In contrast, also correct classifications and good results 
are forgotten by the algorithm. However, the impact is small, which shows the 
results at time sample r = 2,3,4, where the green line is only slightly above the 
blue line. 


—— without forgetting factor 
— with forgetting factor 


wrong predictions (Yo) 


time steps 


Figure 6.22: Results of vehicle combination: Scenario 4. Change of the road conditions and 
simulated with and without forgetting of previous classifications. 


Conclusion and discussion 


With the presented methods to identify trajectories from different vehicles 
for each road road segment, an automatic annotating of sensor data with the 
label of the ground truth is possible. Hereby, the label of road segments, 
known or predicted from already trained vehicles, is transferred to new and 
untrained vehicles. For this purpose, the trajectories of multiple vehicles 
are compared and road segments, driven by various vehicles, are identified. 
Problems can occur with poor GPS signal. However, this can be avoided by 
applying an integrated navigation system, which is based on the sensor data 
fusion of GPS and MEMS-IMU to provide an accurate position of the vehicle 
[193, 194]. Modern vehicles are already equipped with integrated navigation 
systems. Furthermore, the difference of the time duration for obtaining the 
ground truth of a specific road segment and the time when a new vehicle drives 
this road segments and receives the label should be small. In the meantime, 
the ground truth could be changed, e.g. the condition of the road segment is 
significantly decreased or maintained. The investigation of methods to combine 
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the prediction of various vehicles for specific road segments show, that the 
combination based on the precision matrix of the classifier outperforms the 
combination based on majority vote. Various extreme scenarios are simulated, 
e.g. a sudden change of the ground truth or wrong predictions of a vehicle, 
to quantify the impact on the classification results. Moreover, methods to 
overcome these extreme situations are presented. The combined processing is 
mainly performed in simulation. The next steps are to apply these methods on 
a fleet of vehicles. 
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This work presents novel methods to comprehensively and automatically esti- 
mate the road condition with vehicle based sensors. To acquire the data from 
the sensors, a novel measurement system was developed [195]. The instal- 
lation and operating process of the measurement system requires almost no 
manual intervention and the data can be automatically transferred to a central 
data base. A road unevenness index (RUI) is estimated with an inertial sensor 
in the vehicle body and road roughness index (RRI) with an acoustic sensor in 
the tire cavity. The estimation of the indices are based on the measurement of 
the vibration of the vehicle and tire with the mentioned sensors. It was shown 
from experiments under controlled conditions, that the vibrations are not only 
dependent on road roughness or unevenness but also on influences from the 
environment or vehicle operation, especially velocity. To decrease the effect of 
these influences, the indices are estimated for specific bins or intervals, which 
are constructed based on the main influences. The comparison with ground 
truth data suggest that the estimators conform with the actual road condition. 
The estimators of various vehicles can be combined and visualized on maps. 
Furthermore, the trend of the road condition can be identified with a robust 
regression over time for each road segment to be investigated. 


Supervised learning data processing is introduced for the inertial and acoustic 
sensor to classify road features and road roughness. The major novelties 
include a feature selection with multivariate analysis of variances (MANOVA) 
to reduce the complexity and avoid over-fitting of the classifier. Two classifiers 
for the inertial sensor data are developed to estimate both, road surface type 
and road attributes. The prediction of the methods were observed and the 
results suggest a good performance. 


Moreover, methods are introduced to realize an automatic transmission of 
labels to new measurement vehicles to train their classifier. With the same 
methods, road segments and lanes can be identified which are driven by various 
vehicles and methods to combine the output of these vehicles are investigated 
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to improve the prediction rate. Overall, the methods are combined in a Matlab 
toolbox and graphic user interface for evaluation of data sets with alternative 
parametrization by non-experts. 


Summarized, this work achieves the following goals 


e demonstration of current practice and state-of-the-art to monitor the road 
condition, 

e comprehensive literature review of research in automatic and mobile 
road condition monitoring, 

e development of a novel concept to monitor the road condition, which 
incorporates an inertial sensor and acoustic sensor, 

e development of an extended simulation approach to collect data with a 
great variety of vehicle parameters for road condition estimation, 

e investigation of effects, which influence the measuring data, 

e investigation of vehicle and data processing parameters, which influence 
the classification results, 

e development of a novel data processing chain to process inertial sensor 
and acoustic sensor data for road condition estimation, 

e development of a method to combine estimations from different vehicles 
and to automatically train new vehicles, 

e acquisition of data sets to test the data processing methods, 

e implementation of all concepts and methods, which results in a novel 
measuring device and a user friendly toolbox to process the acquired 
data. 


The developed methods can improve or partially substitute the current practice 
for road infrastructure evaluation to increase the automation rate and to reduce 
the costs and response rate. The results of the developed methods show 
that the inertial sensor and acoustic sensor represent a very good option to 
collect information of the tire road contact at low costs and over wide area. 
The data processing methods can run on electronic control units (ECUs) of 
modern vehicles, which serve as sensor platforms. Moreover, as the developed 
system is of modular design, the number and type of sensors and sensor 
modality can be varied. Lastly, a vehicle fleet can be equipped with the 
presented system as it is to estimate the condition of the road infrastructure. 
For example, the German Federal Environment Agency currently deploys seven 
measuring devices, developed in this work, and applies the presented methods, 
to automatically and comprehensively monitor the road condition. 
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